There is no secret to why a global outage occurred on Friday 19th July 2024, it was due to a breakdown in best practice IT management and all parties in the chain must bear some responsibility.
Release Management
While most larger organizations implement release management processes before deploying new software builds, one area that has fallen through the cracks is that of security applications that require frequent updates. Despite major outages in the past, auto-updating of such applications has become commonplace due to the perception that it is required to detect the never-ending array of new vulnerabilities that crop up out of the blue. Validating such updates before deploying has somehow been in the too hard basket for most companies. It is obvious that this is a risky approach.
The Crowdstrike outage showed us, that an uncontrolled update, untested in your environment can have a disastrous impact; a global outbreak of the dreaded blue screen of death, with every affected device requiring a manual reboot in safe mode. For some large organizations, operations stopped altogether. Crowdstrike took responsibility early and released recovery instructions but their lapse in release management due diligence was shocking. Incredibly, the CEO, Greg Kurtz, was CTO at McAfee when a 2010 update caused a key Windows file to be deleted leading to a similar event.
Hopefully the Crowdstrike incident will force the adoption of a more measured approach. If the industry won’t do it, then we can expect additional regulations or harsher penalties. Release management is after all a core SDLC process and regulators in industries like Financial Services and Critical Infrastructure already have coverage over this topic. It will be interesting to see how regulators react, as in theory severe penalties should be handed out for such a basic failure in process.
Release management is a core responsibility for every organization.
Ever Decreasing Circles
The erosion of best practice is part of an ongoing cycle that some companies seem destined to follow. In the early 2000s, everyone was invested in “security at the core” and selected products accordingly. When there weren’t any incidents for a while, cheaper corner-cutting products started to gain favor. Then there was an outage, and everything went back full circle.
We have seen it in trade surveillance, business continuity, trading regulations, and outsourcing. A big event like the LIBOR Scandal, Hurricane Sandy or COVID happens, and best practice becomes popular again.
Contact Us and let JP Reis to help define your strategy and defend your selections. The correct solution is normally cheaper than the consequences of a poor one.
What Can Companies Do?
It’s rare for a company to release an update that is so fundamentally flawed and untested. However, corporate app users cannot afford to be complacent. New software releases often bring improvements, but they can also introduce bugs, vulnerabilities or compatibility issues that can jeopardize infrastructure and operations.
The first step is to truly implement a release management process for every system and application, prioritizing the most widely deployed. No application update should be released to end users and production systems without prior testing being signed off. Stability of a company’s tech stack is one of the key challenges of our time and skimping on how changes are managed leads to increased instability. It is easier said than done, but it’s such a core principle that needs to be adhered to.
Product managers need to stay informed about updates and potential issues and carefully analyze new release notes for potential impacts, security fixes, and known issues. Conduct a thorough risk assessment to understand the potential consequences of upgrading or not upgrading and proceed accordingly. In the case of security patches, the risk associated with auto-updates still exists, however, the risk of not deploying them means they should be top priority within a controlled process.
Release QA Automation
Implement automation as part of your release management process by scheduling updates in test environments and recording the results. Despite being wary of auto-updates for critical applications, a level of automation is advisable to maintain speed and flexibility. Automated testing is mainstream in 2024 for many organizations and this needs to become more widespread.
Testing and Validation Environments
Depending on your scale and criticality, maintain the appropriate DEV/UAT/SIT environments for release testing. Perform comprehensive testing programs including functional, performance, security, and compatibility checks. Include regression testing to ensure that new releases don’t introduce issues to existing functionalities.
If possible, in line with the risk assessment, perform a phased rollout to minimize risks. Continuously monitor system performance and behavior after the release and put a clear rollback plan in place in case of unexpected issues.
Change Management and Communication
Establish a formal change management process to control and document changes. Maintain open communication with stakeholders about the release process, potential impacts, and mitigation plans. Develop a comprehensive incident response plan to address any unforeseen issues and exceptions. Plans need to address less critical apps, third party integrations and other dependencies.
Take Action
The correct approach pays off in the long term and many companies, including ours, were largely unaffected by the outage. None of the principles described above are new, although selecting the right automation services and applying best practice across the organization can be challenging, especially when people are looking to cut costs and there are no big incidents for a while. It takes expertise and strength of character to maintain focus. Contact us to find out how we can protect your business.
