SN 984: CrowdStruck - Crowdstrike, Cellebrite, More Entrust

Last week saw a major security incident involving CrowdStrike, which caused an unprecedented IT outage affecting 8.5 million Windows machines. Steve Gibson, our security expert, breaks down the details of this incident, explaining that a simple memory pointer mistake in a CrowdStrike agent led to the issue. This mistake caused the agent to attempt to load data from non-existent memory, leading to a crash that Windows couldn't recover from. The only solution was to visit each affected machine and fix the issue manually, resulting in significant costs. While this incident dominated the headlines, other important news emerged, including the FBI's reported break-in into a smartphone belonging to a deceased would-be assassin of Trump, and Cisco's discovery of a serious remote authentication vulnerability with a maximum CVSS 10.0 severity rating. Additionally, Google announced that cookies would not be washed away as previously anticipated, and Steve shared some interesting anecdotes from his weekly podcast mailings.

Password-protected Android phones can be unlocked by the FBI using assistance from companies like Celebrate, but iPhones running certain versions of iOS remain a challenge for these phone hacking companies. Recently, the FBI encountered an issue unlocking a Samsung Android phone belonging to a would-be assassin, leading them to seek help from Celebrate. Within hours, Celebrate provided the FBI with the necessary software to unlock the phone and gain access to the shooter's social media, browsing history, and other data. However, documents leaked from Celebrate indicate that they cannot unlock iPhones running iOS 17.4 and later, making it difficult for law enforcement to access critical information from these devices. The documents also suggest that most iPhones running iOS 17.1 to 17.3.1 are not yet unlockable, but the company is working on a solution. The Verge reported that phone hacking companies are overstating their capabilities, and it remains to be seen if they will be able to keep up with the constant evolution of phone security. In a separate incident, Cisco disclosed a maximum security vulnerability in their Smart Software Manager on-premises devices, allowing remote unauthenticated users to change passwords at will. This underscores the importance of keeping software up to date and the potential risks associated with on-premises management systems.

Entrust, a certificate authority (CA) whose certificates were no longer trusted by Google due to past misissuances, is working to regain trust by partnering with SSL.com to issue certificates under SSL.com's auspices. Entrust's certificates issued before October 31, 2024, will continue to be trusted, but new ones will come from SSL.com. This arrangement allows Entrust to retain customer relationships and potentially regain the trust of the CA browser community. The deal is a win-win for both parties, but it feels like a temporary solution for Entrust as they work to improve their operations and rehabilitate their image. Additionally, Entrust is using Things Canary, a modern honeypot solution, to detect and alert on unauthorized access to their network.

Canary is a security tool that helps organizations detect unauthorized access to their networks by simulating deceptive honeypot devices. Users can choose a profile, register for monitoring and notifications, and wait for attackers to reveal themselves. Canary offers various pricing plans, including a two-month money-back guarantee. Cookies, once intended for maintaining session state, have been abused for tracking and profiling users across the web. Google's attempt to eliminate third-party cookies and introduce privacy-preserving alternatives has faced resistance, leading to the continuation of third-party cookies in Chrome. Canary is an effective solution for identifying insider threats and other network intrusions, offering peace of mind for businesses looking to strengthen their security posture.

Defaults matter and can significantly impact user experience and privacy online. Apple's Safari browser has a default setting that allows third-party cookies, which many users unknowingly turn off due to privacy concerns. However, first-party cookies are necessary for website functionality. The EU's cookie regulations have contributed to the belief that all cookies are bad, but that's not the case. Google, which relies heavily on third-party cookies for its business, has implemented an API that requires their use. The discussion also touched on email delivery issues, including false positives from antivirus software and invisible text that triggered spam filters. Overall, it's essential to be aware of the impact of defaults and the importance of understanding the nuances of privacy and security online.

Building a data warehouse in the cloud requires careful consideration, as each unique use case may require specific resources. This principle is reflected in Snowflake's architecture, which allows for multiple virtual warehouses within one system. Additionally, the importance of security was highlighted through an anonymous listener's experience with CDK's dealership outage, which led to the need for manual intervention and updating interface credentials. The use of resource hashes, while convenient, can pose a security risk if external resources are updated without notice. Ultimately, a balance must be struck between convenience and security. Lastly, a discussion about a potential denial-of-service attack using a Linux daemon to block incoming connections raised the importance of considering the potential downsides of active blocking, such as the possibility of creating a denial-of-service condition if an attacker spoofs their source IP address.

Proper testing and preparation are crucial in preventing large-scale IT disasters. The CrowdStrike event, which caused widespread outages and affected thousands of workstations and servers, highlights the importance of having procedures and policies in place to mitigate risks. Many listeners shared their experiences of long recovery times and the devastating impact on their organizations. Some suggested that CrowdStrike, as a continually updated software, should have prioritized testing and pilot groups before rolling out updates to all users. Others proposed the use of trickle-down updates and automated reporting to identify and address issues promptly. Overall, the incident underscores the need for a well-thought-out and proactive approach to IT management.

The widespread use of enterprise security software, in this case CrowdStrike Falcon, can lead to major IT outages affecting critical infrastructure and services in various sectors such as transportation, finance, healthcare, and government. The faulty update caused millions of Windows systems to crash with a blue screen of death, leading to significant disruptions and potential financial losses. The recovery process was lengthy and required manual intervention on each affected system. Similar incidents have occurred with other security vendors in the past. While some argue that security software can be a cause of such outages, it's important to note that they are not unique or new, and most impact only a subset of users. The nature of security software, which needs to run inside the operating system kernel, makes it essential but also prone to causing system crashes.

CrowdStrike, a leading cybersecurity company, experienced a major outage caused by a faulty update to its Falcon endpoint detection and response (EDR) software. The update, intended to protect against new malware techniques, instead triggered a logic error that caused Windows systems to crash with a BSOD. The error spread globally as the update was delivered to CrowdStrike's vast customer base, including many Fortune 500 companies and government departments. The incident has significant implications beyond the technical details and global outages, potentially leading to political repercussions and debates over security vendors' access to the Windows kernel. Microsoft, which holds a dominant market position in the OS market, could be under pressure to kick security vendors out of the kernel, but this move could have negative consequences for system visibility and security. CrowdStrike, founded in 2011, is a cybersecurity giant with over 29,000 corporate customers and has been instrumental in detecting and uncovering major malware campaigns. The company's Falcon software, which runs on millions of computers worldwide, provides crucial visibility into cyber threats. Despite the recent outage, CrowdStrike's role in protecting against cyber threats remains vital.

Even the most advanced cybersecurity companies like CrowdStrike can experience software vulnerabilities leading to system crashes. In this case, an update to a configuration file caused a logic error, resulting in a blue screen of death (BSOD) on Windows systems. The file in question, named "Primer," was part of CrowdStrike's behavioral protection mechanisms and controlled how Falcon evaluated named pipe execution on Windows systems. The update aimed to target newly observed malicious named pipes used in cyber attacks but instead caused a system crash due to a logic error. Big ID, a data security posture management solution, can help organizations manage and reduce data risks, including uncovering dark data and automating data retention. It seamlessly integrates with existing tech stacks and provides advanced AI models to reduce risk and accelerate time to insight. The US Army and other major enterprises have adopted Big ID for its powerful capabilities.

A recent IT incident caused approximately 8.5 million Windows-based machines to crash due to a flaw in CrowdStrike's boot start driver. This driver, designed to enhance Windows' anti-malware defenses, instead caused a blue screen of death error by attempting to load a non-existent memory location. The root cause analysis is ongoing, but it appears that a named pipe API was involved. Microsoft's own documentation identifies a boot start driver as a critical driver required for Windows to start, and in this case, CrowdStrike's driver was given this designation. The competition between good and bad actors to establish the first foothold in an operating system was highlighted, as the entity that arrives first can protect itself from subsequent events. The incident underscores the importance of operating system integrity and the potential consequences of a flawed boot start driver. The investigation is ongoing, with updates to be provided as more information becomes available.

Despite CrowdStrike's advanced security measures, a significant software issue went undetected and led to a massive global issue. The cause of this issue is still unclear, with theories suggesting it could be due to unexpected failure paths or distribution issues. CrowdStrike's silence on the matter has raised concerns, leading some to speculate about potential repercussions and the possibility of insufficient testing. The industry as a whole is working to reverse engineer the issue and determine the root cause, but it's unlikely that CrowdStrike will provide a detailed explanation. Microsoft's past experience with a similar, albeit less catastrophic, event highlights the potential for unlikely failures in even the most robust systems. Additionally, the distribution of unsigned files and potential vulnerabilities in interpreters are potential contributing factors. Regardless of the cause, the incident underscores the importance of thorough testing and the potential risks of running code in Ring Zero.

An exception occurs when attempting to read from non-existent memory, which can potentially lead to serious issues such as crashes or even system reboots into a different operating system copy. This can be particularly problematic for third-party drivers running at a low level, as seen in the recent Microsoft Windows incident. While some suggested disabling problematic drivers on boot, this may not be a feasible solution for all cases. The root cause of the issue remains unclear, with speculation ranging from incompetence to cyber attacks. End users should exercise caution when running antivirus software due to its necessary low-level permissions, which can potentially create vulnerabilities. Microsoft's hardware abstraction layer is designed to mitigate these risks by allowing communication with hardware at a safer level. Ultimately, the importance of understanding and addressing these issues lies in their potential impact on both individuals and organizations, as well as the broader implications for the future of technology.