Podcast Summary
The Illusion of Predicting Future: Despite common belief, predicting future isn't about being unusually smart or knowledgeable, but rather understanding the pitfalls in our thinking and practicing better habits of mind.
Our fascination with predictions and the people who make them can often lead us astray. According to Shankar Vedanta from Hidden Brain, many pundits and experts have a false sense of confidence in their abilities to predict the future. Phil Tatlock, a psychologist at the University of Pennsylvania, even went so far as to compare their predictions to those of dart-throwing chimpanzees. However, there are individuals who are exceptionally good at predicting the future, and Tatlock's research in his book "Super Forecasting" explores how we can learn from them to improve our own forecasting skills. It's important to remember that predicting the future isn't about being unusually smart or knowledgeable, but rather understanding the pitfalls in the way we think and practicing better habits of mind. So the next time you find yourself listening to pundits on television, keep in mind that their predictions may not be as accurate as they claim. Instead, focus on learning from those who have honed their forecasting skills through practice and discipline.
The Dangers of Overconfidence and Relying on Intuition: Overconfidence and intuition can lead to incorrect forecasts and unnecessary actions. Always ensure solid evidence before making predictions.
Overconfidence and relying too heavily on intuition can lead to incorrect forecasts and unnecessary actions. This was exemplified in the story of Archie Cochran, a pioneer of evidence-based medicine who, despite his skepticism, made a costly mistake by undergoing unnecessary surgery based on the appearance of a renowned specialist rather than waiting for the biopsy results. This phenomenon is known as the "bait and switch heuristic." Cochran's experience served as a lesson for him to develop techniques to help doctors guard against intuitive errors and challenge conventional wisdom. Despite his unpopularity during his time, Cochran's contributions are now widely recognized and celebrated. The lesson here for political pundits and anyone making predictions is to be aware of the dangers of overconfidence and to ensure that they have solid evidence before making forecasts.
Impact of biases on accurate predictions: Biases can impact accurate predictions, hindsight bias and glorification of correct predictions can obscure role of luck, untrained individuals can perform as well as experts, consistent performance over time is a reliable indicator of skill
Making accurate predictions is challenging, and our biases can significantly impact our ability to make correct forecasts. Archie Cochrane, a researcher with a mischievous streak, demonstrated this when he presented reversed results of a heart attack study to his colleagues, causing outrage before revealing the true findings of longer survival for patients going home. Hindsight bias and the glorification of correct predictions can obscure the role of luck in successful forecasting. A government forecasting tournament revealed that untrained individuals could perform as well or better than experts, challenging assumptions about the necessity of specialized knowledge. Ultimately, consistent performance over time is a reliable indicator of skill in prediction.
The Mindset of a Successful Forecaster: Believe forecasting is a skill, cultivate curiosity, learn from experience, update beliefs frequently, and leverage historical data to improve forecasting abilities.
The key to becoming a successful forecaster lies in the mindset of believing that forecasting is a skill that can be cultivated and is worth investing time and effort into. Superforecasters, like Bill Flack, possess a unique curiosity and willingness to learn from experience. They understand that being well-calibrated and able to distinguish degrees of uncertainty is crucial for accurate forecasting. Superforecasters also update their beliefs frequently and learn from historical data to make informed predictions. As Aaron Brown, a former world-class poker player and current chief risk officer of AQR, puts it, "You can tell the difference between a world-class poker player and a talented amateur by their ability to distinguish more degrees of 'maybe'." By adopting a growth mindset and focusing on historical data, individuals can improve their forecasting abilities and join the ranks of the superforecasters.
Consider the base rate before making predictions: Start with the average outcome before factoring in specific details to make more accurate predictions
When making predictions or forecasts, it's beneficial to start with an outside perspective, considering the base rate or average outcome in a particular situation, before factoring in specific details. This approach, known as the outside view, is more plausible and realistic than relying solely on personal observations and assumptions (inside view). For instance, when estimating the likelihood of a couple's divorce based on their appearance and happiness at a wedding, starting with the outside view involves looking at the overall divorce rate for their demographic group before considering any unique factors about the couple. This method puts you in a more accurate ballpark and allows for more informed adjustments based on new information. Our natural tendency to rely on the inside view and think in stories may hinder our ability to make accurate predictions, but practicing the outside view can help us appreciate the value of statistical data and the gradual impact of new information.
The Conjunction Fallacy and Storytelling Impact Our Forecasting: The Conjunction Fallacy can make us overlook base rates and overestimate specific, story-like events, leading to inaccurate forecasts. To improve, start with the base rate and use gradations of 'maybe' with probability estimates.
Our tendency to be drawn into stories can negatively impact our ability to make accurate predictions. For instance, when presented with separate scenarios, people often fail to recognize that one event is a subset of a more general one. This is known as the conjunction fallacy. For example, people may find it easier to imagine a specific flood in California caused by an earthquake dam failure, rather than a more abstract flood anywhere in North America. This can lead to inaccurate forecasts, as people overestimate the likelihood of specific, story-like events. To improve forecasting skills, it's essential to start with the base rate or outside-in view and be aware of the risks of storytelling. Additionally, professionals use many gradations of "maybe" and attach specific probability estimates to their predictions. However, even with these techniques, it's challenging to distinguish between those who are good at making predictions due to skill versus luck. An illustrative example is the business school coin toss exercise, where a person who wins multiple coin tosses in a row may appear skilled but could just be lucky.
Balancing accuracy and insightfulness in forecasting tournaments: Future forecasting tournaments should aim to balance both accuracy and insightfulness to provide valuable insights for decision-making.
The first generation of forecasting tournaments focused primarily on improving forecasting accuracy, but there is a need to focus on the insightfulness of the questions as well. The skeptics argue that some investors, like Phil Tetlock, may be anointing lucky individuals as "superforecasters." However, the questions answered in these tournaments were important, even if they were not always the most relevant from a public policy perspective. Some questions may be too hard to forecast with certainty, and it's important to consider whether the goal of superforecasting is to answer simple questions with accuracy or to tackle more complex and significant questions. The next generation of forecasting tournaments should strive to balance both accuracy and insightfulness to provide valuable insights for decision-making.
Balancing the desire for accurate forecasts and social needs: While forecasting tournaments can improve accuracy, they can also be limiting and trivial. It's essential to consider social and psychological factors when engaging in forecasting activities, and adopt a more nuanced approach that acknowledges the complexities of real-life decision making.
While forecasting tournaments can provide valuable insights and improve accuracy for certain questions, they can also become trivial and limited in their forecasting horizon. Moreover, people's emotional needs for reassurance and certainty can make them less receptive to accurate but unsettling predictions. Therefore, it's crucial to strike a balance between the desire for accurate forecasts and the social and psychological goals that people have when they engage in forecasting activities. Additionally, it's important to remember that forecasting tournaments are unique environments that prioritize accuracy above all else, whereas people's real-life decisions are influenced by a multitude of factors. Thus, a more nuanced and holistic approach to forecasting, one that acknowledges the limitations and complexities of the human experience, may be more effective in the long run.
Understanding the reasons behind our beliefs: Recognizing whether our beliefs are evidence-based or driven by social factors can help us navigate different situations effectively.
It's important to be honest with ourselves about the reasons behind our beliefs. While evidence-based beliefs may be crucial in certain contexts, such as forecasting tournaments, people often hold beliefs driven by social factors like fitting in or feeling good about themselves. Recognizing which "game" we're playing can help us navigate different situations more effectively. Phil Tatlock, author of "Super Forecasting," emphasized this during his interview on Hidden Brain. Remember, our team includes Jenny Schmidt, Maggie Pennman, Renee Klar, and Parth Shah, with special thanks to Portia Robinson-Migas from our marketing team. Tune in to Hidden Brain each week on your local public radio station, and connect with us on Facebook, Twitter, and Instagram. I'm Shankar Vedantam, and this is NPR.
