Security experts said CrowdStrike's routine update of its widely used cybersecurity software
which caused clients' computer systems to crash globally on Friday
apparently did not undergo adequate quality checks before it was deployed
The latest version of its Falcon Sensor software was meant to make CrowdStrike clients' systems more secure against hacking by updating the threats it defends against. But faulty code in the update files resulted in one of the most widespread tech outages in recent years for companies using Microsoft's Windows operating system
hospitals and government offices were disrupted
CrowdStrike released information to fix affected systems
but experts said getting them back online would take time as it required manually weeding out the flawed code
the vetting or the sandboxing they do when they look at code
maybe somehow this file was not included in that or slipped through," said Steve Cobb
chief security officer at Security Scorecard
which also had some systems impacted by the issue
Problems came to light quickly after the update was rolled out on Friday
and users posted pictures on social media of computers with blue screens displaying error messages
These are known in the industry as "blue screens of death"
a security researcher who specialises in studying threats against operating systems
said his analysis identified the code responsible for the outage
The update's problem was "in a file that contains either configuration information or signatures"
Such signatures are code that detects specific types of malicious code or malware
"It's very common that security products update their signatures
because they're continually monitoring for new malware and because they want to make sure that their customers are protected from the latest threats," he said
The frequency of updates "is probably the reason why (CrowdStrike) didn't test it as much," he said
It's unclear how that faulty code got into the update and why it wasn't detected before being released to customers
this would have been rolled out to a limited pool first," said John Hammond
principal security researcher at Huntress Labs
"That is a safer approach to avoid a big mess like this."
Other security companies have had similar episodes in the past
McAfee's buggy antivirus update in 2010 stalled hundreds of thousands of computers
But the global impact of this outage reflects CrowdStrike's dominance
Over half of Fortune 500 companies and many government bodies such as the top US cybersecurity agency itself
the Cybersecurity and Infrastructure Security Agency
Explainer - Experts say the cyber outage reveals the risks of an increasingly online world
The global cyber outage will have big ramifications for the IT industry
Emergency Management Minister Mark Mitchell says the CrowdStrike outage has had a limited impact on the health system and many of the services are now coming back online
Some customers were getting "quite agitated" as payment systems went down in the global cyber outage