System Back Online with New Understanding of Internet Fragility

The system is back online, and it's a testament to the tireless efforts of our team who worked around the clock to resolve the issue. The root cause of the problem was a critical failure in our network architecture, which highlighted the fragility of modern internet infrastructure.

Our engineers discovered that a single point of failure in our distributed system led to a cascading effect, causing widespread outages. This was a sobering reminder of the importance of redundancy and fail-safes in our design.

The new understanding of internet fragility has led to a major overhaul of our system, with a focus on building in more robustness and resilience. We're now better equipped to handle unexpected failures and minimize downtime.

The lessons learned from this experience will be invaluable in our future development, ensuring that our system is more reliable and secure for our users.

Upon identifying a major system outage, your initial response should be to assess the scope and impact. Determine which services are down and how many users are affected.

You should communicate quickly with your team and any relevant stakeholders to inform them of the issue. Establish a line of communication for updates and to manage expectations.

To contain the problem and prevent further damage, you may need to shut down certain systems or isolate network segments.

Be clear and concise in your initial response. This is crucial in emergency situations where every second counts.

In a crisis, people are often in shock or panic, and they need clear information to understand what's happening. Research shows that clear communication can reduce anxiety and stress in emergency situations.

Use simple language and avoid jargon. A study found that people are more likely to understand and follow instructions when they're communicated in a clear and simple way.

Provide specific information about what's happening and what people should do next. This will help people feel more in control and prepared to take action.

In a real-life example, a 911 operator's clear and concise instructions helped a caller stay calm and provide vital information about the emergency.

When users report a system outage, it's crucial to start by understanding the nature of the issue. Begin by asking troubleshooting questions in an attempt to gather information.

You should ask yourself if there is an error message, if the issue can be replicated on different devices, if you can check connectivity, and if you can ping the server by hostname and by IP address.

The number of users impacted is also a crucial factor to consider. You need to assess the need for escalation and ask yourself if you need to escalate this to the IT Manager or vendor immediately.

Here are some key questions to ask when assessing the issue:

By answering these initial questions, you can start to narrow down the issue and determine the best course of action for troubleshooting.

Diagnosing an issue requires a systematic approach, especially when dealing with on-premises systems. Start by checking system logs and monitoring tools for any anomalies or error messages that occurred just before the outage.

To diagnose the root cause of the outage, examine recent changes to the system, such as software updates or configuration adjustments, which could have triggered the problem. If necessary, consult with other IT professionals who might have insights into the issue.

Once you've identified the cause, it's time to formulate a recovery plan. This should include step-by-step actions to bring systems back online, prioritizing critical services.

A thorough diagnosis is essential for developing an effective recovery plan. This includes identifying common points of failure, such as loose or disconnected cables, and inspecting for any visible signs of hardware failure.

Here are the key steps to diagnose and recover from a system outage:

Continuous monitoring is crucial to ensure that the fix is permanent and that no secondary issues have arisen as a result of the outage or the recovery process. This also provides peace of mind to users and stakeholders that the systems are functioning correctly.

Water Resources lab staff conduct daily testing at the source and throughout the distribution system for total coliform, E. Coli, and chlorine. This is an increase from the usual 8-10 stations per day, with 35 stations being tested daily on average since Helene.

The water sampling plan was developed in collaboration with the EPA and North Carolina DEQ to ensure the safety of customers. Most sample stations are taps going right into the water lines, with a total of 184 sampling stations throughout the distribution system.

Water Resources' in-house testing is slightly more accurate when the water is clear, so samples are sent to a third-party lab, resulting in a turnaround time of 7-10 days.

To protect human health, the federal government has set primary and secondary maximum contaminant limits for drinking water. Here are the primary limits for some key contaminants:

Note that the levels of iron and manganese in the unfiltered water have slightly exceeded the maximum contaminant levels.

As of December 2 at 12:05 p.m., the City of Asheville's Water Resources Department has resumed billing activities after pausing them due to Hurricane Helene.

The department had paused billing to ensure that all customers had access to potable water, and the boil water notice was lifted on November 18, 2024.

Customers can expect to see a combined utility statement that includes services for water, sewer, stormwater, and sanitation, as applicable.

The Water Consumption Charge for water usage has been changed to $0.00, covering all water usage from the last bill before the storm through the current meter reading.

Sewer Treatment charges will be billed at the regular rate, as the Metropolitan Sewerage District remained operational and treated wastewater on a normal schedule.

Due to the lapse in billing, customers will see their flat fees for other services doubled on their most recent bill, but this will not include a water consumption charge for the billing period impacted by the hurricane.

The City of Asheville will not assess any delinquent fees for utility statements until after March 1, 2025, and customers can contact Water Customer Services at 251-1122 with billing questions or to request additional time to pay the bill.

Payment plans will be available to customers who request them, and regular rates will resume beginning with the next billing cycle.

If you're concerned about lead exposure, especially if you're pregnant, breastfeeding, or have children under 6, it's essential to note that external use like showering, dishwashing, and washing clothes are not at risk for lead exposure.

However, if you have concerns, it's recommended to contact your healthcare provider.

The City of Asheville is taking steps to prevent similar issues in the future, and customers can expect to see regular rates resume beginning with the next billing cycle.

To enable access during a critical outage, it's essential to grant temporary elevated privileges to key personnel. This means giving them the necessary rights to fix the issue, but only temporarily.

As Adnanali Khan, Sr. Support Queue Manager at Snowflake, emphasizes, "To resolve an urgent system outage efficiently, ensure your team has the necessary access rights to the systems they need." This may involve granting temporary elevated privileges, which should be carefully managed to maintain security protocols.

In fact, Mohamed Nacim Herga, IT Systems Manager at Arabian Drilling Company, stresses the importance of maintaining a record of who has been given access to what, not only for accountability but also to reverse these changes once the situation is resolved.

Here are some key points to keep in mind when enabling access during a critical outage:

By following these steps, you can ensure that your team has the necessary access rights to resolve the outage efficiently, while maintaining security and accountability.

Diagnosing an issue requires a systematic approach, especially when dealing with on-premises systems. It's essential to start troubleshooting at the physical layer in the server room if the system isn't reachable via the network.

Charles Duoto, a Broadcast Engineer and IT Admin, suggests asking yourself troubleshooting questions like "Is the system powered on?" and "Is the system plugged into the network?" Inspect for any loose or disconnected cables, as these can often be the root cause of the problem.

Check if the switch connected to the system is powered on, and examine recent changes to the system, such as software updates or configuration adjustments, which could have triggered the issue. A thorough diagnosis is essential for developing an effective recovery plan.

If necessary, consult with other IT professionals who might have insights into the issue. A systematic approach will help you identify common points of failure and get to the root cause of the problem.

A recovery plan is essential to get your systems back online quickly and efficiently. This plan should include step-by-step actions to prioritize critical services and mitigate data loss.

You should document each step of your recovery process to aid in post-mortem analysis and future response strategies. This will help you identify areas for improvement and refine your recovery procedures.

A recovery plan should also consider your backup and disaster recovery protocols to ensure that you can restore data or failover to a secondary system if needed.

Here are some key elements to include in your recovery plan:

Conducting a post-outage review is crucial to identify what went wrong and how it was handled. This review should result in actionable recommendations to prevent future outages.

Gather input from everyone involved in the response effort and document lessons learned. This includes examining whether existing procedures were adequate and identifying areas for improvement.

CrowdStrike, for example, conducted a post-outage review after their recent outage. They found that an issue with a Falcon content update for Windows Hosts caused the crash.

To identify resources, start by gathering data on the affected systems and services during the outage. This includes server logs, network traffic analysis, and any other relevant metrics.

The data collection process should be thorough and well-documented, including timestamps and user interactions. This will help you pinpoint the root cause of the issue.

Server logs can provide valuable insights into system performance and errors leading up to the outage. For example, a sudden spike in error messages may indicate a software issue.

Network traffic analysis can help you identify communication breakdowns between systems. This may reveal issues with data transfer or synchronization between servers.

User interactions, such as error reports and feedback, can also provide critical information about the outage's impact on customers. Analyzing this data can help you understand the human side of the outage and identify areas for improvement.

By gathering and analyzing these resources, you can develop a comprehensive understanding of the outage and make informed decisions about how to prevent similar incidents in the future.

A post-outage review is a crucial step to learn from mistakes and prevent future outages. It's essential to conduct a thorough review to identify what went wrong and how it was handled.

Gather input from everyone involved in the response effort, including team members, clients, and stakeholders. Document lessons learned and examine whether existing procedures were adequate.

CrowdStrike's recent outage is a great example of this. According to an alert sent by CrowdStrike to its clients, the company's "Falcon Sensor" software caused Microsoft Windows to crash and display a blue screen.

The review should result in actionable recommendations to prevent future outages or to handle them more effectively if they do occur. This will help strengthen your systems and your response capabilities.

Anne Neuberger, deputy national security advisor for cyber and emerging technology, confirmed that the incident did not appear to be related to a cyber attack. She noted that it's believed to be an IT-related patch issue with a Falcon content update for Windows Hosts.

The internet infrastructure fragility was exposed during the recent outage, and it's a wake-up call for us all.

The White House was investigating the issue between 6 and 7 a.m., and they were in touch with global banks that were reporting service disruptions.

The outage had a ripple effect, with Portland's emergency services being impacted, and the city's mayor declaring an emergency.

Delta Air Lines resumed some flight departures, but there were still issues with the airline's systems.

The Federal Communications Commission got involved, working with federal agencies to provide assistance and determine the extent of the outage.

President Joe Biden was briefed on the situation, and his team was in touch with CrowdStrike and impacted entities.

The Department of Homeland Security said they were working to fully assess and address system outages, and they were in touch with CrowdStrike, Microsoft, and other partners.

The outage highlights the fragility of the world's core internet infrastructure, as noted by Ciaran Martin, Professor at Oxford University's Blavatnik School of Government.

CrowdStrike's CEO apologized for the inconvenience and disruption, stating that the issue was not a cyber attack, and customers' data remained protected.