Stories Câu chuyện nghề

Memorable moments, unexpected challenges and lessons from a developer’s life.

One Wrong Command, Four Hours: The Amazon S3 Story of 2017

On 28 February 2017, a command with one mistyped input removed more servers than intended from Amazon's S3 storage service and slowed or stopped many internet services for about four hours. This retelling follows AWS's own post-event summary and draws out the safeguards a tool with the power to remove should have.

Oct 29, 20269 min read

#incident
#Amazon S3
#AWS
#postmortem
#operations tooling
#blast radius

StoriesSlides

The Day an Engineer Deleted the Wrong Data: The GitLab 2017 Story

On 31 January 2017 a GitLab engineer ran a delete command on the wrong server and lost about 300 GB of data. Then five layers of backups all failed. This retelling follows GitLab's own postmortem and ends with a checklist for the minute before you type a delete command.

Oct 29, 202610 min read

#incident
#backups
#deleted by mistake
#GitLab
#postmortem
#operations