Podcast Summary
John Ayanidis: A Polymath Expert in Meta-Research: John Ayanidis, a Stanford professor with a background in mathematics, medicine, and epidemiology, challenges the accuracy of most published research in biomedical field using mathematics and advocates for rigorous evaluation of scientific findings to improve human lives
John Ayanidis, a polymath and Stanford University professor, is a leading expert in meta-research, which involves studying the credibility of scientific research. He has a unique background in mathematics, medicine, and epidemiology, and his work includes a groundbreaking paper that challenges the accuracy of most published research in the biomedical field. Ayanidis is passionate about using mathematics to transform our approach to scientific questions and improve human lives through medicine. He grew up in Greece with a love for mathematics and later pursued a career in medicine due to its potential to save lives. Despite his diverse interests, he sees medicine and mathematics as complementary fields. Throughout the podcast episode, Ayanidis shares insights on his research and the importance of rigorously evaluating scientific findings.
The Importance of Clinical Exposure and Research Background in Reliable Clinical Research: Most clinical research is unreliable due to biases and poor methodology, emphasizing the need for rigorous scientific methods and evidence-based medicine.
The reliability of clinical research depends on rigorous scientific methods and quantitative tools. The speaker, a physician-scientist, emphasized the importance of both clinical exposure and research background. He grew up with parents who were physician-scientists and was exposed to their clinical work and research. He completed medical school in Greece and furthered his training in the US, including at Harvard and Tufts. During his training, he was influenced by several individuals, including a professor of epidemiology in Greece and physician-scientists at Harvard and Tufts. These mentors instilled in him the importance of evidence-based medicine, which he encountered in the early 1990s. He was particularly inspired by Tom Chalmers, who was a pioneer in evidence-based medicine and the first to design a randomized trial and perform meta-analyses. The speaker's work came onto the interlocutor's radar in 2005 with a provocative paper in the open-source journal, PLoS Medicine, titled "Why Most Published Clinical Research is Untrue: A Systematic Review of Systematic Reviews." The paper argued that most clinical research is unreliable due to various biases and poor methodology. This call for greater rigor in clinical research continues to be a significant issue in medical science.
Assessing the reliability of scientific research: A mathematical model was proposed to calculate the probability that a statistically significant result is not a red herring, considering the discovery potential, study power, and presence of bias in a field.
The reliability of scientific research is a significant concern due to the high percentage of published papers that go uncited and the prevalence of biases and unreliable results. A mathematical model was proposed to help assess the likelihood of a non-null effect in research, taking into account the field's discovery potential, study power, and the presence of bias. The model aims to calculate the probability that a statistically significant result is not a red herring, considering the prior chances of discoveries in various fields and the power of the studies to detect true effects. Additionally, the model considers the impact of bias, which can make null results appear significant, and the interplay of biases among the vast scientific workforce. This framework provides a valuable tool for evaluating the credibility of research findings and addressing the challenges of ensuring reliable evidence in scientific decision-making.
Statistical significance vs Clinical significance: While statistical significance refers to the likelihood of observing a result, clinical significance determines its practical impact. The default threshold for statistical significance is 0.05, but it may not indicate clinically significant findings. Understanding both is crucial for accurate research interpretation.
The concept of statistical significance, often denoted by a p-value, is not the same as clinical significance. While statistical significance refers to the likelihood of observing a result as extreme as the one obtained given an infinite number of trials, clinical significance refers to the practical impact or importance of a finding. In most research fields, including medicine, the default threshold for statistical significance is 0.05, but there is a growing consensus that this threshold should be more stringent. However, a low p-value does not necessarily indicate a clinically significant finding, as it could be due to bias or other factors. The misconceptions around statistical significance have led to what some call the "statistics wars," with some advocating for abandoning the term altogether and focusing solely on effect sizes and uncertainty. Despite this, it is essential to maintain a nuanced understanding of both statistical and clinical significance to interpret research findings accurately.
Underpowering studies can lead to false positives and exaggerated results: Ensure studies are adequately powered to minimize errors and maximize clinically significant findings
While statistical significance and pre-specified hypotheses are important in some types of research, underpowering studies can lead to false positives and exaggerated results. The scientific community has been operating in underpowered environments due to limited resources, competition, and incentives to publish seemingly statistically significant results. This can result in a high number of false positives or exaggerated estimates of the magnitude of signals. On the other hand, overpowering studies can provide more accurate results, but they also come with their own challenges. It is crucial to find a balance and ensure that studies are adequately powered to minimize errors and maximize clinically significant findings.
Addressing challenges in nutritional epidemiology: Nutritional epidemiology should adopt large-scale studies, collaboration, and transparency from genetics research to improve accuracy and reduce bias.
While the era of big data and genetics research has brought about significant advancements, it also presents new challenges such as measuring bias and ensuring statistical significance. This issue is particularly prevalent in genetics research, which has seen a cultural shift towards transparency, large-scale studies, and collaboration. However, in fields like nutritional epidemiology, which relies on less accurate measurement tools and is fraught with strong opinions and beliefs, self-policing and cultural shifts have been slower to occur. The lack of accurate measurement tools and the presence of strong opinions in nutritional epidemiology contribute to the propagation of subpar information. To address these challenges, nutritional epidemiology should adopt the genetics approach of large-scale studies, collaboration, and transparency to maximize power and standardization.
Genetics and Nutrition Research Progress at Different Paces: Genetics research advances faster with objective methods, while nutrition research faces criticism for sub-optimal methods and fraudulent studies. Approach nutrition research with caution, but there's solid evidence for many observational associations.
While both genetics and nutrition research aim to improve our understanding of health and wellbeing, they have progressed at different paces due to various scientific, cultural, and methodological reasons. Genetics research has advanced faster and become more objective, while nutrition research has been criticized for its sub-optimal methods and the prevalence of poorly done or even fraudulent studies. Austin Bradford Hill, a pioneer in epidemiology, would likely have been cautious about the claims made based on observational associations in nutrition research, recognizing that none of his criteria are bulletproof. Despite the challenges, there is solid evidence for many observational associations in nutrition, such as the link between tobacco use and lung cancer. However, it's essential to approach the field with caution and continue working towards more rigorous research practices.
Strong and consistent odd ratios between smoking and health outcomes: Smoking's impact on health outcomes, such as cancer and cardiovascular disease, is significant and consistent, with odds ratios often in the double digits, contrasting typical odds ratios in epidemiology.
The relationship between smoking and various health outcomes, particularly cancer and cardiovascular disease, is characterized by strong and consistent odd ratios, often in the range of 10, 20, and 30. This is in stark contrast to what is commonly seen in epidemiology today, where odds ratios of 1.01 are more typical. The strength and consistency of the smoking-health outcome relationship are so impressive that other criteria, such as specificity, may not be as important. However, in the field of nutritional epidemiology, where signals are often weaker and less consistent, other criteria may play a more significant role. For example, experimental plausibility can be important, and large randomized trials have been conducted to test nutritional hypotheses. While some of these trials have shown benefits, many have not, leading some to question the validity of observational signals in nutrition. Ultimately, a balanced approach is necessary, recognizing that both observational and experimental evidence have their strengths and limitations.
Understanding dietary efficacy in real life vs. controlled studies: While controlled studies provide insights, real-world effectiveness matters most for diet and nutrition research.
While it's important to understand what diet may be theoretically better, what truly matters is its effectiveness in real-world scenarios with real people. However, there's still a role for studying dietary efficacy under controlled circumstances, such as in clinical trials or Mendelian randomization studies. These approaches can provide valuable insights, but it's crucial to interpret the results cautiously due to the limitations of these methods. Additionally, advancements in data analysis, like exposure-wide association studies, can help us better understand the complex relationships between various nutrients and health outcomes. Ultimately, a more comprehensive and transparent approach to studying diet and nutrition is necessary to make meaningful progress in this field. John's 2005 paper on the frequency of valid scientific publications and his 2012 paper on the systematic cookbook review serve as reminders of the challenges and importance of this ongoing research.
Study on common ingredients and cancer risk sparks controversy: Researcher calls for better data, evidence, and unbiased synthesis in nutritional epidemiology, acknowledges challenges in overcoming deeply held beliefs and institutional biases.
The study selected 50 common ingredients from a cookbook at random and found that nearly all of them had published studies associating them with cancer risk. However, the paper's reception in the nutritional epidemiology community was mixed, with some criticisms coming from concerns about transparency and potential cherry-picking of data. The researcher emphasized the importance of generating better data, evidence, and synthesis in a more unbiased way, regardless of personal beliefs or opinions. They also acknowledged the challenge of overcoming deeply held beliefs in scientific fields. Despite disagreements, the researcher expressed a willingness to see whatever result emerges from a more rigorous and unbiased approach. The researcher also highlighted the importance of large-scale meta-analyses but cautioned against cherry-picking studies based on preconceived beliefs. Institutional biases were also mentioned as a potential issue. Overall, the researcher's goal was to improve the scientific process and generate more reliable evidence in the field of nutritional epidemiology.
Impact of Beliefs on Scientific Research: Beliefs and conflicts can influence research findings, leading to non-reproducible results or incorrect conclusions. Transparency and self-awareness are crucial to minimizing their impact.
While human nature and beliefs play a role in scientific research, it's crucial to minimize the chances of fooling ourselves and maintaining transparency about potential conflicts and beliefs. The DB Cooper case serves as an example of how deeply held beliefs can lead individuals to believe they've identified the culprit, even when they're likely wrong. In science, this can lead to non-reproducible results or incorrect conclusions. Research practices are substandard in many scientific fields, with surveys suggesting that while fraud may be uncommon, suboptimal practices are more prevalent. Transparency and self-awareness are essential to minimizing the impact of beliefs and conflicts on research findings.
Addressing questionable research practices: Continuous training, community awareness, and promoting better methods and inference tools are essential to improve research standards and create an environment that supports high-quality, innovative research.
While scientific misconduct may not be common, questionable research practices are prevalent and require attention. Addressing these issues involves continuous training, community awareness, and the promotion of better methods and inference tools. It's essential to remember that the scientific community is vast and diverse, with varying levels of expertise and familiarity with best practices. Rather than cutting back on research, the focus should be on improving research standards and creating an environment that supports and incentivizes high-quality, innovative research. This approach will ultimately lead to a more reliable and trustworthy scientific literature.
Balancing Industry, Government, and Philanthropy in Scientific Research: Industry prioritizes financial returns, limiting their funding for high-risk research. Government funding is essential but often subject to legislative processes and lacks scientific expertise. Philanthropy plays a crucial role in supporting high-risk research, and public funding should focus on testing benefits and harms.
The funding landscape for scientific research is complex and requires a balance between industry, government, and philanthropy. Industry prioritizes financial returns, limiting their risk appetite for discovery. Government funding, while essential, is often subject to legislative processes and lacks scientific expertise, making it a suboptimal place for high-risk research. Philanthropy, with no strings attached funding, plays a crucial role in supporting high-risk research that might not be funded otherwise. However, public funding is also necessary and should be realigned to focus on testing the benefits and harms of products, rather than financing their development. The communication of scientific research to the public is also crucial, with scientists needing to take a more active role in explaining the high-risk nature of research and the importance of funding it. The success stories of high-risk projects like the Manhattan Project and NASA demonstrate that the public is willing to support such endeavors when effectively communicated.
Defending the honesty of science in public communication: Scientists must be transparent and avoid exaggeration when communicating with the public, using lobbying tactics or scientific methods. Complex nutritional research highlights the need for better training and methods to prevent misleading results, and the consequences of inaccurate information can be significant.
As scientists, it's essential for us to defend the honesty and principles of science when communicating with the public. However, the current landscape of media and agendas can make it challenging to be heard above the noise. Two paths forward are using lobbying tactics or sticking to scientific methods. While both have their merits, it's crucial to avoid exaggeration and maintain transparency. The complexity of nutritional epidemiology research highlights the need for better methods and training to prevent misleading results. The consequences of inaccurate information can be significant, and it's our responsibility as scientists to ensure the public receives reliable information. By fostering humility and understanding, we can bridge the gap between scientific knowledge and public awareness.
Compromised randomization process in Predimed study: The importance of rigorously executing clinical trials to maintain their credibility and ensure valid results, even with the high cost and duration involved.
The integrity of a clinical trial is paramount. The Predimed study, which compared a low-fat diet to two Mediterranean diets (one with olive oil and the other with nuts), showed promising results initially. However, it was later discovered that the randomization process had been compromised, leading to a partly randomized and partly observational trial. This significant flaw casts doubt on the credibility of the trial's results, despite its initial clinically relevant and statistically significant findings. The importance of meticulously executing clinical trials, even with the high cost and duration involved, cannot be overstated. This example serves as a reminder of the potential consequences of research that falls short of the gold standard of randomized controlled trials.
Variability in research expertise leads to potential bias and inaccuracies in multi-center clinical trials: The reliability of data in multi-center clinical trials can be compromised due to varying levels of methodological training and expertise among researchers, potentially leading to bias and inaccuracies.
The reliability of data in multi-center clinical trials, particularly those funded by industry, can be compromised due to varying levels of methodological training and expertise among researchers involved. This can lead to bias and inaccuracies, even if the data isn't outright fraudulent. The 2020 COVID-19 pandemic brought forth the importance of this issue when studies on seroprevalence, such as those in Santa Clara and LA County, revealed much higher rates of infection than confirmed cases, challenging prior beliefs about the virus's mortality rate and spread. These findings were met with both excitement and controversy due to the politically charged environment surrounding the pandemic.
Controversy over COVID-19 infection fatality rate study: A study on COVID-19 infection fatality rate faced criticism during the pandemic's early stages. Criticism led to improvements and the study's accuracy was later validated. Importance of separating science from politics and basing findings on data was emphasized.
During the early stages of the COVID-19 pandemic, there was significant controversy surrounding a study estimating the infection fatality rate. The study, which was later validated by subsequent research, showed that the virus was rapidly and widely spreading, and the infection fatality rate varied greatly depending on the population. The author expressed gratitude for the criticism, which helped improve the paper, and acknowledged that some of the backlash was due to the toxic political environment at the time. He emphasized the importance of keeping science separate from politics and urged that findings be based on data and scientific evidence, rather than political ideology. Despite some misconceptions and personal attacks, the study ultimately proved to be accurate and influential in understanding the nature of the virus.
Attacks on Scientists During Crises: Consequences and Solutions: Protecting scientists and promoting a culture of tolerance towards scientific findings is crucial during crises to prevent silencing and hindrance of scientific progress.
The attacks on scientists and their research during times of crisis, such as the ongoing COVID-19 pandemic, can have devastating consequences not only for the individuals involved but also for the scientific community as a whole. These attacks, which can come from various political ideologies, can silence scientists and hinder the progress of science. It is crucial that we protect scientists and allow them to do their work without fear of smear campaigns or threats to themselves and their families. The scientific process involves making mistakes, correcting them, and updating beliefs with new information. In politics, however, changing one's mind is often seen as a weakness. It is essential to promote a culture of tolerance and understanding towards scientific findings, regardless of political affiliations. The recent surge in public interest in science due to the pandemic presents an opportunity to communicate more science and improve science education, but it also poses challenges in the form of lobbying forces and political turmoil.
Renowned Scientist John Ioannidis remains optimistic about future discoveries: Renowned scientist John Ioannidis emphasizes the importance of staying curious and open-minded, looking forward to new discoveries and potential corrections.
Renowned scientist John Ioannidis remains excited and optimistic about the future of science, viewing each new discovery as an opportunity to expand his knowledge and correct any potential errors. Ioannidis emphasizes the importance of remaining curious and open-minded, as there is still so much to learn. Regarding their personal connection, they expressed hope for another dinner together in 2021. The Drive podcast offers a membership program with exclusive content, including comprehensive podcast show notes, monthly AMA episodes, access to a private podcast feed, discounts on recommended products, and more. The podcast is for general informational purposes only and does not constitute medical advice. Conflicts of interest are disclosed on Ioannidis' website.
