a/b testing

Brought to you by Mixpanel—Event analytics that everyone can trust, use, and afford | Round—The private network built by tech leaders for tech leaders | Eppo—Run reliable, impactful experiments

—

Ronny Kohavi, PhD, is a consultant, teacher, and leading expert on the art and science of A/B testing. Previously, Ronny was Vice President and Technical Fellow at Airbnb, Technical Fellow and corporate VP at Microsoft (where he led the Experimentation Platform team), and Director of Data Mining and Personalization at Amazon. He was also honored with a lifetime achievement award by the Experimentation Culture Awards in September 2020 and teaches a popular course on experimentation on Maven. In today’s podcast, we discuss:

• How to foster a culture of experimentation

• How to avoid common pitfalls and misconceptions when running experiments

• His most surprising experiment results

• The critical role of trust in running successful experiments

• When not to A/B test something

• Best practices for helping your tests run faster

• The future of experimentation

—

Enroll in Ronny’s Maven class: Accelerating Innovation with A/B Testing at https://bit.ly/ABClassLenny. Promo code “LENNYAB” will give $500 off the class for the first 10 people to use it.

—

Find the full transcript at: https://www.lennyspodcast.com/the-ultimate-guide-to-ab-testing-ronny-kohavi-airbnb-microsoft-amazon/

—

Where to find Ronny Kohavi:

• Twitter: https://twitter.com/ronnyk

• LinkedIn: https://www.linkedin.com/in/ronnyk/

• Website: http://ai.stanford.edu/~ronnyk/

—

Where to find Lenny:

• Newsletter: https://www.lennysnewsletter.com

• Twitter: https://twitter.com/lennysan

• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/

—

In this episode, we cover:

(00:00) Ronny’s background

(04:29) How one A/B test helped Bing increase revenue by 12%

(09:00) What data says about opening new tabs

(10:34) Small effort, huge gains vs. incremental improvements 

(13:16) Typical fail rates

(15:28) UI resources

(16:53) Institutional learning and the importance of documentation and sharing results

(20:44) Testing incrementally and acting on high-risk, high-reward ideas

(22:38) A failed experiment at Bing on integration with social apps

(24:47) When not to A/B test something

(27:59) Overall evaluation criterion (OEC)

(32:41) Long-term experimentation vs. models

(36:29) The problem with redesigns

(39:31) How Ronny implemented testing at Microsoft

(42:54) The stats on redesigns 

(45:38) Testing at Airbnb

(48:06) Covid’s impact and why testing is more important during times of upheaval 

(50:06) Ronny’s book, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

(51:45) The importance of trust

(55:25) Sample ratio mismatch and other signs your experiment is flawed

(1:00:44) Twyman’s law

(1:02:14) P-value

(1:06:27) Getting started running experiments

(1:07:43) How to shift the culture in an org to push for more testing

(1:10:18) Building platforms

(1:12:25) How to improve speed when running experiments

(1:14:09) Lightning round

—

Referenced:

• Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing: https://experimentguide.com/

• Seven rules of thumb for website experimenters: https://exp-platform.com/rules-of-thumb/

• GoodUI: https://goodui.org

• Defaults for A/B testing: http://bit.ly/CH2022Kohavi

• Ronny’s LinkedIn post about A/B testing for startups: https://www.linkedin.com/posts/ronnyk_abtesting-experimentguide-statisticalpower-activity-6982142843297423360-Bc2U

• Sanchan Saxena on Lenny’s Podcast: https://www.lennyspodcast.com/sanchan-saxena-vp-of-product-at-coinbase-on-the-inside-story-of-how-airbnb-made-it-through-covid-what-he8217s-learned-from-brian-chesky-brian-armstrong-and-kevin-systrom-much-more/

• Optimizely: https://www.optimizely.com/

• Optimizely was statistically naive: https://analythical.com/blog/optimizely-got-me-fired

• SRM: https://www.linkedin.com/posts/ronnyk_seat-belt-wikipedia-activity-6917959519310401536-jV97

• SRM checker: http://bit.ly/srmCheck

• Twyman’s law: http://bit.ly/twymanLaw

• “What’s a p-value” question: http://bit.ly/ABTestingIntuitionBusters

• Fisher’s method: https://en.wikipedia.org/wiki/Fisher%27s_method

• Evolving experimentation: https://exp-platform.com/Documents/2017-05%20ICSE2017_EvolutionOfExP.pdf

• CUPED for variance reduction/increased sensitivity: http://bit.ly/expCUPED

• Ronny’s recommended books: https://bit.ly/BestBooksRonnyk

• Chernobyl on HBO: https://www.hbo.com/chernobyl

• Blink cameras: https://blinkforhome.com/

• Narrative not PowerPoint: https://exp-platform.com/narrative-not-powerpoint/

—

Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@lennyrachitsky.com.

—

Lenny may be an investor in the companies discussed.

Get full access to Lenny's Newsletter at www.lennysnewsletter.com/subscribe

Brought to you by Public—Invest in stocks, treasuries, crypto, and more | Eppo—Run reliable, impactful experiments | Writer—Generative AI for the enterprise

—

Laura Schaffer is the brand-new VP of Growth at Amplitude. Prior to this role, she spent over 10 years leading product management and growth teams at Twilio, Bandwidth, and Rapid. In today’s episode, we talk about the role of experimentation and data in growth, and Laura shares stories of big wins from her time leading growth teams. She explains how customer insights helped her uplevel her career and how she (surprisingly) thinks about qualitative versus quantitative data. We wrap up our conversation by discussing where the best ideas come from and what you need to know if you’re selling to developers.

Find the full transcript here: https://www.lennyspodcast.com/career-frameworks-ab-testing-mistakes-counterintuitive-onboarding-tips-selling-to-developers-laura-schaffer-vp-of-growth-at-amplitude/#transcript

Where to find Laura Schaffer:

• LinkedIn: https://www.linkedin.com/in/lauraschaffer/

Where to find Lenny:

• Newsletter: https://www.lennysnewsletter.com

• Twitter: https://twitter.com/lennysan

• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/

Referenced:

• Elena Verna on Lenny’s Podcast: https://www.lennyspodcast.com/elena-verna-on-how-b2b-growth-is-changing-product-led-growth-product-led-sales-why-you-should-go-freemium-not-trial-what-features-to-make-free-and-much-more/

• Bandwidth: https://www.bandwidth.com/

• Twilio: https://ahoy.twilio.com/

• Jeff Lawson on LinkedIn: https://www.linkedin.com/in/jeffiel/

• The Surprising Power of Online Experiments: https://hbr.org/2017/09/the-surprising-power-of-online-experiments

• Reforge: https://www.reforge.com/

• Online Experimentation at Microsoft: https://ai.stanford.edu/~ronnyk/ExPThinkWeek2009Public.pdf

• The Simple Path to Wealth: Your Road Map to Financial Independence and a Rich, Free Life: https://www.amazon.com/Simple-Path-Wealth-financial-independence/dp/1533667926

• Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones: https://www.amazon.com/Atomic-Habits-Proven-Build-Break/dp/0735211299/

• James Clear on The Tim Ferriss Show: https://podcasts.apple.com/us/podcast/648-james-clear-atomic-habits-simple-strategies-for/id863897795?i=1000592431628

• The Great British Baking Show on PBS: https://www.pbs.org/food/shows/great-british-baking-show/

• Hotjar: https://www.hotjar.com/

• Amplitude: https://amplitude.com/

• Builder: https://www.builder.io/

• ChatGPT: https://chat.openai.com/

• Lenny Bot: https://www.lennybot.com/

• Segment: https://segment.com/

• Senior Growth PM, Monetization, at Amplitude: https://boards.greenhouse.io/amplitude/jobs/6636704002

• Lead Growth PM at Builder: https://boards.greenhouse.io/builder/jobs/4814755004?gh_src=30cfda2d4us

• Growth PM at Rapid: https://jobs.lever.co/rapidapi/8d2611d1-6463-4919-9817-31f61e730831

In this episode, we cover:

(00:00) Laura’s background

(04:15) How to carve your own career path, and an example from Bandwidth

(05:50) Laura’s career growth framework

(10:18) The value of customer insights

(12:25) The “voice of the customer” report

(16:14) Leaning into your strengths

(18:16) The experiment that shifted the way Laura thinks about friction

(20:20) Questions that improved Twilio’s onboarding and conversion rate

(28:53) Thinking about the psyche of your users

(31:26) The hot dog analogy for burying “scary stuff”

(33:58) Why it’s better to be iterative and why experiments fail

(36:21) Saving money by validating fast

(41:58) Where the best ideas come from

(49:51) Experimentation lessons

(52:54) The amount of time a growth team needs to be successful 

(54:43) The big change at Twilio that led to tens of millions of dollars

(58:41) The need for both PLG and enterprise, and how Amplitude plans to tap into PLG

(1:05:42) What it’s like to serve developers

(1:11:16) Lightning round

Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@lennyrachitsky.com.

Get full access to Lenny's Newsletter at www.lennysnewsletter.com/subscribe