CrowdStrike, the US-based cybersecurity firm, caused a global outage on July 19 after an update resulted in Windows laptops and desktops crashing and getting stuck in a boot loop. The outage lasted multiple hours affecting different sectors including airlines, healthcare, IT, and more. After fixing the issue, the company published a post-incident report highlighting that its artificial intelligence (AI) system dubbed ‘Falcon sensor’ caused an error. Now, the company has published a detailed report after conducting an external review to highlight what exactly went wrong.
CrowdStrike Publishes External Review Report
In a report titled ‘External Technical Root Cause Analysis — Channel File 291’, the cybersecurity firm said it found that the Falcon sensor deployed an erroneous template type string which affected Windows interprocess communication (IPC) mechanisms.
As per CrowdStrike, Falcon runs machine-learning models that automatically identify and remediate the latest and advanced threats from bad actors. Right before the July 19 outage, the detection functionality pushed a new “template type” to millions of computers of customers’ Falcon installations in version 7.11.
However, this is where things went wrong. The report highlighted that the IPC template type had defined 21 input parameter fields but “the integration code that invoked the Content Interpreter with Channel File 291’s Template Instances supplied only 20 input values to match against.” This mismatch is usually not a concern since so far the AI system has never picked an input outside the given 20.
But on that day, the sensor asked to inspect template type 21. Since there was no corresponding integration code relating to it, the attempt to access the 21st input parameter created an out-of-bounds memory error and resulted in a system crash.
Highlighting steps for mitigation, the report claimed that CrowdStrike developed a patch for the Sensor Content Compiler that validates the number of inputs provided by a Template Type. This went into production on July 27. The firm said that it has also focused on increased testing and validation before pushing an update. Further, it has also stated that all future updates will be rolled out in a phased manner to minimize any potential error.
Notably, no details about the external vendors who conducted the review were provided.