GitLab.com melts down after wrong directory deleted, backups fail

Emotional, Burning, Unlimited Tuned Laboratory



Upstart said it had outgrown the cloud – now five out of five restore tools have failed

Source-code hub GitLab.com is in meltdown after experiencing data loss as a result of what it has suddenly discovered are ineffectual backups.

On Tuesday evening, Pacific Time, the startup issued a sobering series of tweets we’ve listed below. Behind the scenes, a tired sysadmin, working late at night in the Netherlands, had accidentally deleted a directory on the wrong server during a frustrating database replication process: he wiped a folder containing 300GB of live production data that was due to be replicated.