Ding Dong, the Book is Dead! (Facebook is down)

[Syonyk]

Someone screwed up their DNS and one can only hope it stays screwed up.

[Syonyk]

Oh… oh man.

From a Reddit thread…

As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that’s that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC).

There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified.

Part of this is also due to lower staffing in data centers due to pandemic measures.

They didn’t leave backup access to the peering routers? I know some other companies still have (or had, and used) dial up modems for “The world has caught fire and we need to get in” access to their big gateway routers to datacenters…

[Syonyk]

https://twitter.com/sheeraf/status/1445099150316503057

(in case this gets removed)

Was just on phone with someone who works for FB who described employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren’t working to access doors.

[Canem]

Their access control system is tied to literally all the same server[s] that the actual websites run on? Why would… you know what nevermind. I’m not actually surprised.

[Syonyk]

Relevant reading:

https://bowaggoner.com/writeups/robust.html

Relevant explanation:

Understanding How Facebook Disappeared from the Internet

[bombcar]

We’re gonna see more of this as companies continue to integrate “single pane of glass” interfaces to their networks - which allow for “singe pain in ass” failures unlike any we’ve seen before.