We (GNU Savannah and the FSF) value transparency, so here are the details on this page.

Recap

There's been a SQL SELECT injection leading to a leak of unsalted MD5 account passwords, some of them discovered through online passwords recovery services, leading in turn to project membership and admin access, used for vandalism on the 'www' project that backs www.gnu.org.

www.gnu.org normally serves static pages, but PHP was found enabled, so crackers installed a reverse shell there as well.

Root access is separately maintained and no evidence of root access was found.

No other exploitable SQL injection has been found in Savane so far, thanks to the SQL parametrization work done back in 2007.

To be safe, we've reinstalled the system and restored the data from a trusted backup, November 23th circa 12:00 GMT.

Counter-measures

  • Crack analysis before re-enabling any service
  • SQL injection fix and code audit before re-enabling the web front-end
  • Removed all passwords (users and system) and sessions
  • Use crypt's SHA-512 for passwords, and phpass's entropy code for salt
  • Enforced password strength (through passwdqc)
  • Added logs analysis reporting tool that keeps us informed of SQL injection attacks
  • Disabled PHP entirely at www.gnu.org
  • Upgraded friend website gna.org to our version of Savane

In progress:

  • Auditing changes between the 23th and the 27th to see what was committed (no code commits found so far)

Thanks to FSF sysadmins Bernie Innocenti and Ward Vandewege, Savannah Hackers Sylvain Beucler, Brian Gough, Michael J. Flickinger, Jim Meyering; Openwall hacker Solar Designer; Savannah users for their support.

Timeline

  • 2010/11/24 21:30 UTC: SQL SELECT injection attack originated from Tbilisi, Georgia, access to user encrypted passwords
  • 2010/11/24 21:27 UTC: one Savannah admin password cracked, account compromised
  • 2010/11/26 16:02 UTC: cracker gained membership to the www project
  • 2010/11/26 23:51 UTC: cracker tested commit to the www CVS repository
  • 2010/11/27 00:51 UTC: cracker defaced www.gnu.org
  • 2010/11/27 01:35 UTC: cracker committed a reverse shell using unexpectedly enabled PHP support
  • 2010/11/27 01:36 UTC: notification of the intrusion
  • 2010/11/27 01:37 UTC: website restored
  • 2010/11/27 04:42 UTC: emergency fix to Savane code (unknowing that an admin account was still compromised)
  • 2010/11/27 19:05 UTC: new cracker activity on www.gnu.org - we shutdown the machines
  • 2010/11/27 21:35 UTC: reinstalled www.gnu.org
  • 2010/11/29 15:23 UTC: reinstalled Savannah machines to be safe
  • 2010/11/29 21:30 UTC: access to the base host restored, extracting incremental backup from the 23th
  • 2010/11/29 23:30 UTC: finished diagnosing original attack
  • 2010/11/30 12:30 UTC: data transfers in progress
  • 2010/11/30 13:30 UTC: read-only access to source repositories
  • 2010/11/30 14:30 UTC: write access to source repositories
  • 2010/11/30 16:30 UTC: data transfers finished
  • 2010/11/30 18:00 UTC: access to downloads and GNU Arch
  • 2010/11/30 21:00 UTC: audited code and found no other SQL injection
  • 2010/11/30 22:30 UTC: found trace of earlier attack on Nov 23th 04:00
  • 2010/11/30 22:45 UTC: stopped write access
  • 2010/11/30 23:45 UTC: found trace of earlier read-only SQL injections as back as January, but none with actual account cracking
  • 2010/12/01 00:55 UTC: after fishing through logs, it appears that there was no other account cracking
  • 2010/12/01 11:00 UTC: restored write access
  • 2010/12/02 08:02 UTC: web front-end improved and re-enabled

Old

This is how Sylvain restored Savannah after the crack.

  • ☑ Put services online using backup, except for password-based ones (e.g. the web interface)
  • ☑ Bring back web interface
  • ☑ Reset passwords
  • ☑ Fix SQL injection and look for potential others
  • ☑ Implement crypt support (like /etc/shadow, strong and LDAP-compatible) hashes, with SHA-512
  • ☑ Implement salt generation with good entropy
  • ☑ Implement password strength enforcement
  • [/] Implement Solar's suggestions (TODO)
  • [/] Implement logs-checking reporting tool (TODO: better init.d support)
  • [/] Implement on-login password rehash for MD5 and improperly salted SHA-512
  • Implemented rehashing for improperly salted hashes.
  • ☑ Implement banner to warn people about changing password (on login pages or all pages?)
  • ☑ Change TLS/https keys
  • ☑ move back dot files on colonialone.fsf.org
  • [/] restore @savannah.gnu.org aliases replication from colo (restored user syncaliases, now check if it works)
  • ☑ restore user svadmin -> not restoring it since it can access nothing in the new xen setup
  • ☐ Drop users ward/peobo/bernie?
  • ☑ Change 2 failed disks
  • [/] savannah-backup:
  • ☑ enable backup cron
  • ☐ add rsnapshot to keep history locally (rdiff seems to be too long to extract backups); coordinate with sysadmin@gnu.org so they backup the latest backup online