data-driven decision making

Send Everyday AI and Jordan a text message

AI + all your data = good move or bad move? Sure, AI can do data crunching wonders when we pair the two together. But, what challenges and conflicts should we be aware of before doing so? Aakash Indurkhya, Head of Product at Virtualitics, joins us to discuss AI and data analysis.

Newsletter: Sign up for our free daily newsletter
More on this Episode: Episode Page
Join the discussion: Ask Jordan and Aakash questions on AI and data

Related Episodes:
Ep 145: NVIDIA Leader Talks GenAI + Data: Unlocking new ways to interact with our world
Ep 142: AI is Changing Data Analysis: Insider Tips

Upcoming Episodes: Check out the upcoming Everyday AI Livestream lineup
Website: YourEverydayAI.com
Email The Show: info@youreverydayai.com
Connect with Jordan on LinkedIn

Topics Covered in This Episode:
1. Generative AI Technology and Data Visualization
2. Role of AI in Data Analysis
3. Statistics Paradox and Data Collection Challenges
4. Importance of Hands-On Experience and Collaboration

Timestamps:
01:30 Daily AI news
04:30 About Aakash and Virtualitics
08:51 Stress of data utilization despite its value.
10:39 Data layers may disagree, causing decision paralysis.
15:33 Generative AI aids human-driven data analytics.
17:38 RAG concept crucial for tracing data decisions.
24:06 Beginning data analysis requires tools, exploration, openness.
26:43 Future of jobs with technology, data and humans.
29:21 Communicate with experts for effective problem-solving.

Keywords:
data in the age of generative AI, large language models, OpenAI model spec, Google DeepMind AlphaFold 3, protein behavior prediction, Zapier Central, AI bots without coding, Virtualisix, data-driven decisions, industry-specific decision intelligence, data abundance paradox, ChatGPT technology, changes in data collection, Apple, Google, maintenance data for HVAC systems, human role in data analysis, industry-specific expertise, generative AI in data visualization, RAG technology, conversational interfaces in generative AI, sensitive data handling, GPT language models, Virtualitix AI platform, decision intelligence apps, data-driven decision paradox, decision congestion, common language in data analysis, data quality, decision socialization.

Get more out of ChatGPT by learning our PPP method in this live, interactive and free training! Sign up now: https://youreverydayai.com/ppp-registration/

Brought to you by Mixpanel—Event analytics that everyone can trust, use, and afford | Round—The private network built by tech leaders for tech leaders | Eppo—Run reliable, impactful experiments

—

Ronny Kohavi, PhD, is a consultant, teacher, and leading expert on the art and science of A/B testing. Previously, Ronny was Vice President and Technical Fellow at Airbnb, Technical Fellow and corporate VP at Microsoft (where he led the Experimentation Platform team), and Director of Data Mining and Personalization at Amazon. He was also honored with a lifetime achievement award by the Experimentation Culture Awards in September 2020 and teaches a popular course on experimentation on Maven. In today’s podcast, we discuss:

• How to foster a culture of experimentation

• How to avoid common pitfalls and misconceptions when running experiments

• His most surprising experiment results

• The critical role of trust in running successful experiments

• When not to A/B test something

• Best practices for helping your tests run faster

• The future of experimentation

—

Enroll in Ronny’s Maven class: Accelerating Innovation with A/B Testing at https://bit.ly/ABClassLenny. Promo code “LENNYAB” will give $500 off the class for the first 10 people to use it.

—

Find the full transcript at: https://www.lennyspodcast.com/the-ultimate-guide-to-ab-testing-ronny-kohavi-airbnb-microsoft-amazon/

—

Where to find Ronny Kohavi:

• Twitter: https://twitter.com/ronnyk

• LinkedIn: https://www.linkedin.com/in/ronnyk/

• Website: http://ai.stanford.edu/~ronnyk/

—

Where to find Lenny:

• Newsletter: https://www.lennysnewsletter.com

• Twitter: https://twitter.com/lennysan

• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/

—

In this episode, we cover:

(00:00) Ronny’s background

(04:29) How one A/B test helped Bing increase revenue by 12%

(09:00) What data says about opening new tabs

(10:34) Small effort, huge gains vs. incremental improvements 

(13:16) Typical fail rates

(15:28) UI resources

(16:53) Institutional learning and the importance of documentation and sharing results

(20:44) Testing incrementally and acting on high-risk, high-reward ideas

(22:38) A failed experiment at Bing on integration with social apps

(24:47) When not to A/B test something

(27:59) Overall evaluation criterion (OEC)

(32:41) Long-term experimentation vs. models

(36:29) The problem with redesigns

(39:31) How Ronny implemented testing at Microsoft

(42:54) The stats on redesigns 

(45:38) Testing at Airbnb

(48:06) Covid’s impact and why testing is more important during times of upheaval 

(50:06) Ronny’s book, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

(51:45) The importance of trust

(55:25) Sample ratio mismatch and other signs your experiment is flawed

(1:00:44) Twyman’s law

(1:02:14) P-value

(1:06:27) Getting started running experiments

(1:07:43) How to shift the culture in an org to push for more testing

(1:10:18) Building platforms

(1:12:25) How to improve speed when running experiments

(1:14:09) Lightning round

—

Referenced:

• Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing: https://experimentguide.com/

• Seven rules of thumb for website experimenters: https://exp-platform.com/rules-of-thumb/

• GoodUI: https://goodui.org

• Defaults for A/B testing: http://bit.ly/CH2022Kohavi

• Ronny’s LinkedIn post about A/B testing for startups: https://www.linkedin.com/posts/ronnyk_abtesting-experimentguide-statisticalpower-activity-6982142843297423360-Bc2U

• Sanchan Saxena on Lenny’s Podcast: https://www.lennyspodcast.com/sanchan-saxena-vp-of-product-at-coinbase-on-the-inside-story-of-how-airbnb-made-it-through-covid-what-he8217s-learned-from-brian-chesky-brian-armstrong-and-kevin-systrom-much-more/

• Optimizely: https://www.optimizely.com/

• Optimizely was statistically naive: https://analythical.com/blog/optimizely-got-me-fired

• SRM: https://www.linkedin.com/posts/ronnyk_seat-belt-wikipedia-activity-6917959519310401536-jV97

• SRM checker: http://bit.ly/srmCheck

• Twyman’s law: http://bit.ly/twymanLaw

• “What’s a p-value” question: http://bit.ly/ABTestingIntuitionBusters

• Fisher’s method: https://en.wikipedia.org/wiki/Fisher%27s_method

• Evolving experimentation: https://exp-platform.com/Documents/2017-05%20ICSE2017_EvolutionOfExP.pdf

• CUPED for variance reduction/increased sensitivity: http://bit.ly/expCUPED

• Ronny’s recommended books: https://bit.ly/BestBooksRonnyk

• Chernobyl on HBO: https://www.hbo.com/chernobyl

• Blink cameras: https://blinkforhome.com/

• Narrative not PowerPoint: https://exp-platform.com/narrative-not-powerpoint/

—

Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@lennyrachitsky.com.

—

Lenny may be an investor in the companies discussed.

Anu Hariharan and Gustaf Alstromer are partners at YC.

This episode is a follow-up to Anu's Growth Guide and the first Growth Office Hours.

The YC podcast is hosted by Craig Cannon.