#346 - Rob Reid - How To Avoid Destroying Humanity

Our fascination with existential risks, which could lead to the end of civilization, may stem from our evolutionary past where the leaders of our ancestors' clans had to consider potential threats to their entire group for survival. However, despite our fascination with these risks, we often remain blind to them due to their newness and the fact that there were no plausible steps for civilization destruction before the mid-1950s. Now, one of the most significant existential risks we face today is synthetic biology, which could potentially lead to the creation of deadly organisms. Another risk is artificial general intelligence, which could surpass human intelligence and potentially pose a threat to our existence. We should learn from the COVID-19 crisis and invest in preparing for and mitigating these risks. Rob Reid, an entrepreneur, podcaster, and author, believes that we should continue technological development but prioritize safety measures and ethical considerations. Overall, it's crucial to acknowledge and address these existential risks to ensure the long-term survival and progress of humanity.

The existence of nuclear weapons in the late fifties led to a new level of existential risk for humanity, which has been a concern for the last 60 years. However, the attention given to this risk is still not enough compared to what it should be. The concept of existential risk is a relatively new development, and the amount of attention it receives today is significantly more than it was 15 years ago. The attention given to climate change, which has been a concern for longer, has grown exponentially due to compounding awareness and the development of large industries and economic interests around it. To prevent existential risks, it is crucial to continue spreading and increasing awareness, as policymakers will likely be the last to recognize and address these threats.

Throughout history, there have been numerous close calls between nuclear-armed nations that could have led to catastrophic consequences. Two notable incidents were during the Cuban missile crisis when Russian submarines were threatened by American depth charges, and during the early 1980s when a Russian facility mistakenly detected an incoming American missile attack. In both cases, the decisions of individual men to question orders and not launch nuclear weapons prevented escalation and potential disaster. These incidents underscore the importance of human judgment and the potential for miscommunication or misunderstanding in the context of nuclear weapons. Despite advancements in technology and safeguards, the risk of nuclear conflict still exists.

As we make advances in synthetic biology and technologies like super AI, the potential for catastrophic events is no longer limited to a select few with access to vast resources. Instead, the risk is spreading to a larger population, making it increasingly difficult to prevent potential disasters. This is a shift from the Cold War era, where billions of dollars were spent to prevent a few people from pressing a proverbial "red button." With synbio, for instance, the rapid improvement of tools and methodologies means that complex and Nobel-worthy work will soon be accessible to high school students and lower-budget companies. This democratization of dangerous technologies poses a significant challenge, as it's impossible to monitor and prevent all potential misuses. The stakes are higher than ever, and the consequences of a mistake or intentional misuse could be catastrophic. This privatization and democratization of the apocalypse underscores the importance of addressing these risks and ensuring that safeguards are in place to mitigate potential dangers.

The absence of nuclear weapons during the geopolitical tensions between superpowers in the 20th century might have led to even more devastating conventional wars. The decision to avoid nuclear war was a public good owned and operated by governments, but the potential creation of super AI in private hands introduces a new level of risk with privatized gains and socialized losses. The financial crisis serves as a reminder of the consequences when individuals or companies take risks with significant societal costs. The potential for catastrophic risks from super AI development underscores the importance of responsible governance and regulation to prevent the creation of existential risks. The nuclear threat acted as a deterrent, but the absence of such a deterrent in the context of super AI development could lead to disastrous consequences.

The democratization of advanced technology through privatization poses a significant risk due to the potential inclusion of numerous agents who may not adhere to safety standards, leading to potentially catastrophic consequences. This risk is amplified in the context of synthetic biology, where a malignant actor could create a deadly and highly transmissible organism. While COVID-19 serves as a reminder of the potential destructive power of even a relatively benign virus, a deliberate release of a more deadly organism could have civilization-ending consequences. The lab leak hypothesis for COVID-19 is plausible, but the virus's relatively low lethality and transmissibility suggest it was not intentionally engineered for maximum destruction. Instead, it may have been released as a warning or a coordination problem to prepare the world for future engineered pandemics. However, if an engineered pandemic were intended to cause widespread panic, the perpetrator would likely make it clear that it was engineered to maximize fear and chaos.

The transmissibility, asymptomatic period, and lethality of a virus can determine its potential impact on society. The speaker mentions measles, SARS, h5m1 flu, and tuberculosis as examples of viruses with varying levels of these factors. The speaker also discusses the danger of lab leaks and the difficulty in preventing them, using the anthrax attacks post-9/11 as an example. The most dangerous virus in history could have the lethality of h5m1 flu, the transmissibility of measles, and a long incubation period like tuberculosis. The speaker emphasizes the importance of public health practices and research in mitigating the impact of such viruses.

The infamous anthrax letters sent after 9/11, which caused widespread fear and panic, were not as securely contained as one might have assumed, even by a highly security-conscious nation like the United States. The anthrax spores, which originated from a US army lab, managed to reach the office of the Senate majority leader, despite the country being on high alert following the terrorist attacks. This incident highlights the potential vulnerabilities of even the most secure labs and the importance of robust biosafety measures to prevent both accidental and malicious leaks. Despite the advanced precautions in place, there have been instances of leaks from high-security biosafety level 4 labs, making it crucial to remain vigilant and continually improve safety protocols.

The potential risks of biotechnology research, particularly gain-of-function research, cannot be ignored. The discussion highlighted the example of h5n1 flu, which was made airborne transmissible in labs, raising concerns about the potential consequences if such research falls into the wrong hands or results in unintended leaks. While proponents argue that such research helps us prepare for potential pandemics, critics caution that the creation of deadly and transmissible pathogens, even in contained lab environments, poses significant risks to global health and safety. The discussion underscored the need for careful consideration and robust safety measures when engaging in such research.

Gain-of-function research, which involves creating or manipulating pathogens to better understand their potential threats and possible countermeasures, is a double-edged sword. While it can yield valuable scientific knowledge, the risks of a lab accident or unintended consequences are significant, especially with highly contagious and potentially deadly viruses like h5n1. The incident involving two labs working on h5n1 gain-of-function research led to a pause in US government funding for several years due to concerns over bio-risks. However, the research was eventually published, and funding was resumed. Despite the potential dangers, gain-of-function research continues, and it's not limited to the US. Scientists argue that the benefits outweigh the risks, but critics question whether those in the field fully grasp the potential consequences of their work. The incident serves as a reminder of the importance of careful consideration and transparency in scientific research, particularly when dealing with potentially dangerous pathogens.

Individuals, even those with expertise and a strong personal incentive, may underestimate the risks they pose to others when making decisions. This was evident in the discussion about an expert scientist's confidence in his ability to avoid a lab leak, despite the potential consequences for society. The Manhattan Project serves as a historical example, where scientists weighed the minuscule risk of igniting the atmosphere against the real possibility of Nazi Germany developing a nuclear weapon first. These decisions, while made with the best intentions, demonstrate the importance of considering the broader implications and potential harm to others when assessing risk.

The decision to develop nuclear weapons during the Manhattan Project was a carefully considered risk, as the potential consequences of a Nazi regime with nuclear capabilities were deemed greater than the risk of an accidental lab mishap. In contrast, gain-of-function research, which involves creating or enhancing viruses in a lab, presents a smaller benefit and a much greater risk, especially when considering the potential for widespread access to dangerous DNA and RNA synthesis capabilities. While there have been efforts to regulate this research, more comprehensive measures are necessary to prevent the creation and dissemination of dangerous pathogens. The Manhattan Project serves as a reminder that sometimes taking calculated risks for the greater good is justifiable, but in the case of gain-of-function research, the potential risks far outweigh the potential benefits. It is crucial that we take serious steps to prevent the creation and distribution of dangerous pathogens, including strengthening regulations and infrastructure to make it difficult for individuals to obtain dangerous DNA and RNA.

While it's important to address potential risks, such as gain of function research, it's equally important to recognize and address more complex and diffuse risks, like deadly viruses in schools. The debate around BSL 3 and BSL 4 labs raises questions about the balance between valuable research and potential risks. While it might be challenging to completely eliminate all apocalyptic microbes, it's crucial to take necessary precautions and build robust safeguards into research tools. The COVID-19 pandemic serves as a stark reminder of our vulnerabilities and the need for improved pandemic preparedness. We have the opportunity to learn from this experience and invest in research and development of universal vaccines and other measures to better protect ourselves against future pandemics. The cost of these measures is relatively small compared to the potential damage caused by a pandemic. By taking a proactive approach, we can harden ourselves against future threats and minimize the impact of potential outbreaks.

Investing in the development of universal vaccines for influenza and other viruses that cause significant economic and health burdens could be a worthwhile investment, despite the high costs and long timelines involved. For instance, the annual cost of flu to the United States is estimated to be $365 billion, and a universal flu vaccine could potentially offer significant relief. While a universal flu vaccine is estimated to cost between $200 million and $2 billion over 10 years, the potential benefits could far outweigh the costs. Similarly, investing in pan-virus vaccines for all virus families that infect and kill humans could be a game-changer, with an estimated cost of $40 billion over 10 years. The potential impact of such investments could be significant, especially in the context of the economic damage caused by the COVID-19 pandemic, which is estimated to have caused $14 trillion in damage to the global economy. However, despite the obvious benefits, there seems to be a lack of concerted effort to invest in such research and development. The same goes for the development of a pan-coronavirus vaccine, which should have started as soon as the COVID-19 pandemic hit but is still not a priority. The implications of a lab leak theory being proven true for the COVID-19 pandemic could be significant, leading to a global ban on gain-of-function research and a renewed focus on preventative measures.

The discussion highlights the importance of transparency and accountability in scientific research, particularly in the context of gain-of-function research in synthetic biology. The speaker believes that a ban on such research could reduce the global risk of synbio being used maliciously, but the opacity surrounding the origins of COVID-19 raises concerns about the trustworthiness of authoritarian governments and the need for thorough investigations. The speaker also questions whether we should consider limiting technological progress to prevent potential existential risks, but ultimately concludes that such a scenario is unlikely and not a practical solution. Instead, the focus should be on improving regulatory frameworks, increasing global awareness, and fostering a culture of openness and transparency in scientific research.

The complex and interconnected nature of technological advancements and global geopolitics presents significant challenges, particularly in the areas of synthetic biology, artificial intelligence, and nanotechnology. The coordination problem of ensuring global cooperation and ethical use of these technologies is a major concern, as the potential for misuse or unintended consequences increases with the democratization and privatization of these technologies. The future is uncertain, and the risks are vast. Therefore, it is essential to adopt a multilayered and adaptive defense strategy, drawing inspiration from our own immune systems. This strategy includes stopping gain-of-function research, creating industry standards for dangerous DNA, and continually adapting to new threats as they emerge. The human flourishing potential of these technologies is significant, particularly in the areas of therapeutics, clean meat production, and ethical considerations. However, the risks and challenges cannot be ignored, and a proactive and agile approach is necessary to mitigate them.

The current system for regulating the production of dangerous nucleic acids, such as synthetic DNA and RNA, is a good start but needs significant improvements. Companies like Twist Bioscience, which specialize in producing long strands of error-corrected nucleic acid, have robust internal safeguards in place to ensure the benign usage of dangerous sequences. However, as the ability to synthesize nucleic acids becomes cheaper and more accessible, the number of orders and the cost of regulation per order are increasing. Voluntary regulation through organizations like the IGSC, which accounts for approximately 80% of the DNA synthesis in the world, is not enough to protect against malicious actors. To address this, it is proposed that regulation become mandatory and that we transition to a distributed model of DNA credit creation, where benchtop DNA synthesizers allow scientists to produce the nucleic acid they need in their own labs. This would reduce the reliance on centralized service bureaus and make regulation more effective and cost-efficient.

As technology advances, particularly in the field of synthetic biology, there is a growing potential for individuals to create dangerous biological agents in their own labs. To mitigate this risk, it's essential that we invest in advanced, distributed printers with robust safety protocols. These printers would make it much harder for individuals to create dangerous DNA sequences, reducing the number of potential threats from a large pool to a small group of experts. The future of these printers could be widespread, even in high schools, making the creation of complex viruses from scratch increasingly obsolete. While this doesn't eliminate the risk entirely, it significantly reduces it by making the skills required to create such agents less accessible. The path to self-improvement and career development often leads away from the creation of dangerous biological agents. Additionally, as printers become more sophisticated, the need for traditional wet lab skills will diminish, making the creation of complex viruses from whole cloth increasingly difficult. While some may find the intricacies of hard science fiction challenging, the potential benefits of advanced distributed printers in synthetic biology are significant in reducing the risk of biological threats.

As the world faces various existential risks, from natural disasters like the moon exploding to man-made threats like malignant AI, it's crucial to consider creating self-sufficient, defended communities as a backup. These communities should be equipped with necessary resources, including books and seeds, and could operate on a rotation system. While ethical questions arise regarding the potential suffering of those living in such communities, the greater good of preserving humanity as a whole may outweigh individual hardships. The conversation also touched upon the relevance of projects like Biosphere 2, which aimed to create an independent biosphere but did not involve bunkering or heavy defense. The prepper movement and wealthy individuals buying up land for potential survival bunkers were also mentioned as related phenomena.

The Biosphere 2 experiment, an ambitious project aimed at creating a self-sufficient ecosystem for human survival, ultimately failed due to agriculture and atmospheric issues. This small-scale endeavor, while charming, lacked the resources and scientific power to sustain its inhabitants for the intended two-year period. However, the idea of creating a self-sufficient ecosystem on a larger scale, such as a national or international effort, is intriguing. While the experience may be intense, similar to those who spend extended periods in extreme environments like Antarctica, the potential risks, including the creation of dangerous forms of matter like ice 9, serve as a reminder of the unknown unknowns and barely known unknowns that come with such ambitious projects. The Biosphere 2 experiment, while failing in its execution, continues to inspire discussions on the possibilities and challenges of creating sustainable ecosystems for human survival.

The existential risk community, including researchers and institutions, need to prioritize making their work more accessible and popular to the public. This is crucial because decisions regarding existential risks are not just made by a select few in cloistered intellectual communities. Public figures, like Greta Thunberg, have shown that a single voice can rally support for important movements. However, the current state of existential risk research is specialized and gated, which means the general public is not engaged or aware. To address this, more public-facing figures and storytellers, such as science fiction writers, should be involved to make the topic more relatable and compelling. The importance of the continued existence and flourishing of the human species cannot be overstated, and it is essential that we involve a broader audience in the conversation. This will not only increase awareness but also put pressure on policymakers to take action. The success of movies like "War Games" and "Dr. Strangelove" in shaping public perception during the Cold War era demonstrates the power of storytelling in influencing society.

Stories, whether it be novels or series, have the power to shift societal perspectives and inoculate against potential threats, such as totalitarianism or superintelligent AI. In the past, works like George Orwell's "1984" and the "Terminator" franchise have played a significant role in shaping public opinion and raising awareness about the dangers of totalitarian regimes and advanced artificial intelligence. However, it's important to remember that even the most intellectually sophisticated individuals are not immune to cognitive biases and the allure of certain ideologies. Therefore, continuing to produce and engage with thought-provoking stories is crucial in promoting a culture that values critical thinking and rational decision-making. The recent series "Next" on NBC, which explores the risks of a misaligned superintelligent AI, is a prime example of how storytelling can help push the cultural conversation forward.

Entertainment and culture can significantly shape public consciousness and push important issues into the mainstream, even before their time. This was evident with movies like "The Terminator" and "1984," which opened people's eyes to the potential dangers of supercomputers and totalitarianism, respectively. For those passionate about existential risks, contributing to the cause can be done in various ways, including supporting research centers, applying public pressure on governments, and even influencing local education systems. Small, organized countries with a history of progressive policies, like Denmark or Estonia, could make a significant impact by making existential risk a priority and allocating substantial resources to the issue. Engaging with local officials and implementing educational programs in schools are also effective ways to spread awareness. Ultimately, the cultural side of raising awareness is just as important as academic discussions, and could potentially be even more powerful and accessible.

Individual actions and awareness, whether it's through policy implementation or personal efforts, can make a significant impact on the environment and other important issues. Rob Reid, a multifaceted individual, has demonstrated this through his various projects, including his podcast where he interviews world-class thinkers, his work on turning a novel into a movie, and his new investing partnership with Chris Anderson to fund startups making the world more resilient. It's inspiring to see how one person can make a difference in multiple areas and inspire others to do the same. Additionally, Reid's dedication to his podcast and music projects showcases the importance of persistence and hard work in achieving success.