Podcast Summary
Impact of research flaws on DevOps: Despite the widespread acceptance of research from Google's Dora team on DevOps practices, recent findings suggest potential flaws and limitations, emphasizing the importance of critical analysis and transparency in research.
The research in the DevOps community, specifically from Google's Dora team, may have flaws that were not initially apparent. Two years ago, Junade Ali studied the impact of developer burnout and found high rates of burnout among software engineers. More recently, he discovered that 75% of software engineers faced retaliation when reporting wrongdoing and 89% of business leaders were concerned about on-time software delivery. However, further investigation revealed issues with the research methods and findings of the Dora team's State of DevOps reports and subsequent frameworks. This issue was reminiscent of the story of Dr. Ansel Keys and the Seven Countries Study, where Dr. Keys' research was widely accepted despite having limitations and incomplete information. Dr. Jenkinson, in his book "Why We Eat, Too Much," criticized Dr. Keys for not presenting the whole truth, even though he did not outright lie. Similarly, the Dora team's research, while technically accurate, may not have presented the complete picture. This discovery underscores the importance of critical analysis and transparency in research, especially in the rapidly evolving field of technology. It also highlights the need for ongoing evaluation and improvement of research methodologies and findings.
Devops research limitations: While Dora's research on devops practices provides valuable insights, its limitations include lack of publicly available data, potential bias from subjective surveys, and the need for further research to establish causal relationship between speed and reliability.
While Dora's research on devops practices, conducted through subjective surveys, provides valuable insights into the relationship between speed and reliability in software development. However, it's important to note that the data underlying these findings is not publicly available, which can limit the ability for external validation and verification of the results. Furthermore, the use of subjective surveys can introduce bias, potentially skewing the data towards certain outcomes. While there may be a correlation between speed and reliability, it's crucial to establish a causal relationship to fully understand the underlying dynamics. Until then, we should interpret Dora's findings with caution and consider the potential limitations of their research methods.
Software development priorities: Agile methodologies prioritize speed and bug fixing, but organizations should consider data security, data accuracy, and preventing serious bugs as top priorities, adopting a risk-proportionate approach.
While agile methodologies like Rapid Application Development (RAD) have been effective in addressing certain software development challenges, such as faster deployment and bug fixing, the importance of these metrics may not align with the priorities of software developers and the public. For instance, during the Toyota unintended acceleration bug incident and the Horizon IT scandal, agile methodologies were adopted, but the focus on speed and bug fixing might not have been the most pressing concern for ensuring data security, data accuracy, and preventing serious bugs. Moreover, business decision-makers prioritize delivering high-quality software on time over fast delivery. Lastly, trust in software engineers and reliability expectations can vary significantly across industries, making a one-size-fits-all approach inappropriate. Instead, organizations should adopt a risk-proportionate and consistent approach with their defined risk appetite.
Following the money in research: While following the money can provide valuable insights in research, it's crucial to maintain a critical perspective and not rely solely on such information.
Following the money can provide valuable insights when evaluating research or investigations. The example given involves Dora, a team that initially did state-of-dev ops reports for Puppet, a company focused on automating IT infrastructure, and now does the same work for Google Cloud. Both Puppet and Google Cloud have a vested interest in developers being able to deploy work quickly. This collaboration has contributed to the world of software engineering by adding empirical evaluation to the process. However, it's crucial to be aware of marketing material's limitations and recognize the potential flaws in such research. In essence, while following the money can offer valuable insights, it's essential to maintain a critical perspective and not rely solely on such information.