Contents:

Free Python Course ➞ Mini-course for beginners and experienced coders. 4 cool projects in the portfolio, live communication with the speaker. Click and find out what you can learn in the course.
Learn moreInternational IT companies sometimes experience unexpected and catastrophic incidents that can lead to extended service downtime, data loss, or, conversely, the unauthorized distribution of information online. In this article, we will look at five major incidents that demonstrated the vulnerability of even the most reliable systems.
Notion blocked due to phishing complaints
On February 12, 2021, from 2 a.m. Moscow time, Notion experienced serious access issues. Users, including many corporate clients, were unable to access their data. This situation has caused significant inconvenience for users who rely on Notion for project management and information storage. Platform loading issues highlight the importance of cloud service reliability for businesses.
In a since-deleted tweet, Notion asked its followers for assistance regarding issues with Name.com, the hosting provider for the notion.so domain. Name.com responded by stating that it is actively working with the domain owners to resolve the issue as quickly as possible. Notion expressed surprise and ironically asked where exactly they should direct the messages.

The Notion team informed TechCrunch about the DNS issues that arose, which are almost resolved. They promised to regularly update the status on their Twitter page.

When you visit a website, your browser sends a request to a DNS server, which translates the domain name into an IP address. This allows your device to find the server hosting the desired service. The process of translating a domain into an IP address is a key step in ensuring fast and efficient access to internet resources. Proper DNS configuration helps improve page loading speed and overall website performance.
Notion registered the notion.so domain through the registrar Name.com. However, domains ending in .so are managed by another company, Hexonet. Hexonet acts as an intermediary, connecting the .so domain with registrars such as Name.com. This interaction resulted in an outage that rendered the Notion platform unavailable.
Name.com spokesperson Jared Evie confirmed the information in an email to TechCrunch.
Hexonet received reports of users creating Notion pages for phishing purposes. As a result, the company temporarily suspended the Notion domain. Following the incident, teams from the three companies joined forces to quickly restore service. We are currently developing new protocols to prevent similar incidents in the future.
GitLab Lost a Database
In January 2017, GitLab experienced a major data loss that affected users for six hours. The incident resulted in the loss of issue reports, pull requests, user data, and comments. However, Git repositories, wikis, and local GitLab instances remained intact. All data that users contributed to the database between 5:20 PM and 11:25 PM UTC was lost. Technical support had to restore the lost information from a backup, which required significant effort and almost manual intervention. This incident highlights the importance of regular backups and robust data management to prevent similar incidents in the future.
On January 31, 2017, at 6:00 PM UTC, GitLab detected an issue with spammers flooding the database by creating numerous snippets. This negatively impacted the stability of the database. The GitLab technical team responded promptly to the situation and began active measures to eliminate the problem.

At 21:00 UTC, a critical situation occurred, as a result of which GitLab blocked all operations records in the database. This led to a complete system shutdown.

The service's technical team successfully resolved the original issue. Their troubleshooting efforts met user expectations, which confirms their professionalism and competence in the field of customer service. You can be sure that the service will continue to provide high-quality support and improve its functionality.
- spammers were identified and blocked by IP address (those who enjoy playing Counter-Strike or Quake on a network from the 2000s will appreciate this);
- a user who was using the repository as a content delivery network (CDN) was removed. Because of this, 47,000 IP addresses logged into the system and nearly crashed the database;
- users were removed for spam.
At 22:00 UTC, GitLab detected serious problems with synchronizing the contents of database copies, which led to a significant lag in replication. This failure was caused by a sharp increase in the volume of write operations, which the database was unable to cope with. This situation requires attention to optimize system operation and prevent similar failures in the future.


The db2 content write fell behind by 4 GB, which resulted in the inability to synchronize db2.cluster with other copies. As a result, the cluster went through the self-cleaning procedure. The problem was exacerbated by db2.cluster's failure to connect to db1 due to a low ax_wal_senders parameter, which limits the number of clients for replication. Attempts to restart PostgreSQL on db1 were unsuccessful, and the system refused to start. After several attempts, PostgreSQL finally started, but replication from db2.cluster failed, and the cluster continued to hang. To resolve this issue, it is necessary to address the replication settings and PostgreSQL configuration parameters.
At 23:00 UTC, a GitLab developer suspected that the database backup was failing due to an empty PostgreSQL directory. In an attempt to resolve the issue, he deleted the directory, but mistakenly did so on the primary server db1.cluster.gitlab.com instead of the replica. By the time he realized his mistake, it was too late: out of 300 GB of data, only about 4.5 GB remained. This incident highlights the importance of careful data handling and the need for regular monitoring of backup processes in database management systems.
The company was forced to shut down gitlab.com and notify users of the issue via Twitter. This temporary suspension was made to address the issue and ensure the security of user data. We understand this may cause inconvenience, and we are making every effort to restore the platform as quickly as possible. Stay tuned to our Twitter account for up-to-date information on the service status.

Of the five backup solutions deployed, none worked. As a result, the GitLab team was only able to restore a backup made six hours earlier. This highlights the importance of reliable backups and the need to regularly test the health of data recovery systems.
Google Cloud ran out of storage
On December 14, 2020, at approximately 2:00 PM Moscow time, a Google Cloud outage occurred. Users experienced problems accessing their documents and were unable to send emails in Gmail. A message appeared on the screen: "Try reloading this page or coming back in a few minutes. We apologize for the inconvenience." The issues lasted for about an hour, causing user outrage and discussion of the incident on social media. Such outages highlight the importance of cloud service reliability for everyday work and communication.
The company reported that an authentication system error was the cause of the service outage.
Today, at 3:47 a.m. Pacific Time (US), Google experienced a temporary authentication system outage that lasted about 45 minutes. The outage was caused by an issue with internal storage quotas. At 4:32 a.m., the issue was resolved and all Google services were back online.

According to The Guardian, the issue was caused by insufficient space allocated in the company's internal systems for authentication services. The system was expected to automatically increase the storage capacity for data, but this did not happen. This highlights the importance of effective resource management in the IT infrastructure, especially in the context of ensuring the reliability and security of services.
The outage at Google caused a number of problems for users. Many experienced difficulties turning on the lights in Google Home devices, holding meetings in Google Meet, and participating in lessons in Google Classroom. The situation with authorization in Slack and other services where access was carried out through a Google account was especially critical. These outages significantly hindered users' work and affected their daily activities.
Facebook* went down for a day
The recent outage of Facebook services is not the first in the company's history. On March 13, 2019, around noon, Facebook, Facebook Messenger, and Instagram experienced global outages that lasted almost the entire day. WhatsApp users also reported difficulties sending photos, while Facebook users saw a notification that the site was temporarily down for maintenance. Such incidents highlight the need for robust systems and technology to keep popular social platforms running smoothly.
Facebook's error reporting platform was temporarily down, making it difficult for users to report service issues. The company then began communicating the situation through Twitter, one of the few major social media platforms not under its control. Around 3:00 PM Moscow time, Facebook confirmed the issue and said it was actively working to resolve it. The company also noted that the outage was not related to DDoS attacks.
On Thursday, Instagram announced on Twitter that all technical issues had been resolved. Facebook later confirmed this information, stating that the service was fully operational again.
Facebook provided an explanation for the outage, which occurred 24 hours ago. Officially, the social network's downfall was caused by changes in its server configuration.
Yesterday, a server configuration change occurred, causing issues with accessing our applications and services for many users. We quickly resolved all issues, and our systems are currently being restored. We apologize for the inconvenience and thank everyone for their patience during this time. We appreciate your loyalty and continue to work to improve the quality of our services.
Facebook to Twitter messaging is an important aspect of social media integration. Sharing content from Facebook to Twitter allows you to expand your audience and increase the visibility of your posts. Using multiple platforms to distribute information helps increase user engagement and improve interaction with your target audience. Sharing Facebook posts on Twitter helps create a unified promotion strategy and contributes to effective social media marketing.
AOL Reveals User Search Query Data
On August 4, 2006, AOL Research published data on 650,000 users of its Internet provider. A compressed text file containing 20 million keywords was posted on one of its websites. This data was initially intended for internal research, but its publication caused a wide public outcry and raised questions about the privacy and security of personal information on the Internet. The AOL case became a significant example of the importance of protecting user data and the need to adhere to ethical standards in processing information.
On August 7, AOL apologized to its users and removed the confidential data from the site. However, by that time, the information had already spread across the Internet. Interestingly, AOL didn't consider this incident a serious problem until a user blogged about it. The situation highlights the importance of protecting personal data and being aware of potential threats in the digital space.
The leaked database contained user search queries for the period from March 1 to May 31, 2006, linked to unique user IDs (user_id). The data included search history and results, as well as the links users clicked. This type of information may be of interest for user behavior analysis and search engine optimization.

Requests associated with random user_ids make it possible to uniquely identify many users. Users often search for their own names or the names of people they know to find out what information about them is available online. As a result, many search queries contain personal information such as names, addresses, Social Security numbers, and other sensitive information. This highlights the importance of protecting personal information and being aware that information posted online may be accessible to others.
In January 2007, Business 2.0 Magazine on CNNMoney included this incident in its list of "101 Dumbest Moments in Business," ranking it #57. This case became a shining example of poor management and strategic mistakes in business, attracting the attention of both professionals and the general public.
Due to the scandal, the company's CTO decided to resign.
Mistakes are an inevitable part of any professional path, but their number can be significantly reduced by thoroughly studying your specialty. On the Skillbox website, in the "Programming" section, you will find courses developed by experienced professionals from leading IT companies. In-depth training under the guidance of professionals will help you quickly master the necessary skills and increase your competitiveness in the labor market.

