No one could have imagined that a social network as large as Facebook could “crash.” Yet, at 10:30 PM yesterday (Vietnam time), Facebook along with several other services like Messenger, Instagram, and WhatsApp all simultaneously ceased operations and became inaccessible. Notably, this issue did not just last for a moment; it extended for nearly 7 hours. This had a serious impact on users worldwide.
So, what exactly happened?
Although Facebook did not disclose the specific reason for its system issues, experts at CloudFlare provided a relatively detailed explanation. Speaking of CloudFlare, it is a company in the content delivery network (CDN) sector, known for its service package that allows websites to operate even in the event of an attack or server problems. Additionally, CloudFlare also operates DNS 1.1.1.1 and the VPN tool WARP+.
BGP: The Core Issue Behind Facebook’s Problems
The essence of the Internet is connecting one computer to another. For instance, when you access Facebook, you are connecting your computer to Facebook’s server to exchange data. The process by which your computer “finds its way” to Facebook’s server is called routing.
BGP, which stands for Border Gateway Protocol, is a mechanism that allows large network systems to exchange routing information. Major routers around the world maintain a regularly updated list of the “routes” available for data packets. Imagine it as a guide map when you’re lost in a large shopping mall.
BGP is like a guide map that you often see in shopping malls.
At the time Facebook crashed, CloudFlare’s team noticed that Facebook’s BGP had been updated to remove routes to this service’s servers. It was like the guide map in the shopping mall had been taken down, and throngs of people arrived but had no idea where to go. In other words, Facebook disappeared from the Internet.
Facebook’s BGP was updated with routing issues at the time this social network experienced disruptions.
The problem with Facebook’s BGP caused the DNS (Domain Name Resolver), or Domain Name System, to fail in its task of translating the domain name facebook.com into an IP address. In case you didn’t know, DNS is responsible for resolving the domain name like facebook.com into the server’s IP address, such as 157.240.211.35. Since Facebook’s BGP lost its routing capabilities, this caused DNS to fail in finding Facebook’s IP address.
So why did Facebook’s BGP crash?
By now, you understand BGP and how it works. But, ultimately, why did Facebook’s BGP encounter issues? Isn’t Facebook supposed to have a very professional team?
On Reddit, a user claiming to work in Facebook’s incident response team provided some “behind-the-scenes” information about this incident.
Specifically, he stated that Facebook’s BGP encountered issues following a configuration change in the system. However, the network system issues at Facebook were so severe that they could not be resolved remotely, requiring admins to be physically present at the datacenter.
However, those who were “on duty” at Facebook’s datacenter did not have enough knowledge and authority to fix the issue. As a result, Facebook’s engineers took longer than usual to restore the company’s services. It is known that Facebook had reduced its on-site team at the datacenter due to the impacts of COVID-19.
After revealing this information, this user’s Reddit account was deleted.