Plan Z

Facebook had a bad day recently. And the next day wasn't too good either.

I've posted a couple of times (here and here) about my "Plan Z."

I've also posted several times (here, here, and here) about WFH risks.

Apparently Facebook doesn't read my blog.

Somebody at Facebook made a mistake. People make mistakes. That will happen.

What happened (or rather what didn't happen) next is the issue.

The Daily Mail had a good recap of the series of problems.

But the repair was delayed, according a purported insider, because of 'lower staffing in data centers due to pandemic measures', ...

There's the "WFH risk." And no Plan A.

Kieron Harding, an IT Infrastructure Engineer at GRC International Group, told DailyMail.com: 'The nature of the problem meant Facebook would have needed network engineers to physically access their BGP routers - and due to the pandemic, some of the data centers quite possibly don't have an engineer based on site, or someone who could have immediately started to work on the problem.'

"Facebook would have needed network engineers to physically access their BGP routers," Facebook didn't have a Plan B.

... the misconfiguration of the BGP also affected Facebook's physical door access systems

Facebook didn't have a Plan C.

You have to have a plan all the way down to Plan Z.

Be prepared.