Podcast summary and Notes on " data-driven experimentation"

Enhancing Gameplay through Niantic's Experiment-Driven Approach

This episode features an interview with Mikey Chang, Group Product Manager at Niantic.

In this episode, Mikey reflects with us on the vast user base and feedback he deals with at Niantic, emphasizing a data-driven approach, A/B testing, and the delicate balance of making incremental changes. Mikey provides insights into the unique dynamics of the gaming industry's business-to-consumer model, where users are both entertained and motivated to exercise, highlighting the challenges he faces in achieving this balance.

Host takeaways:

Kyle:

Ready Up Feature Success Metrics: Mikey emphasized the importance of focusing on measurable outcomes such as retention, revenue, and real-world social impact when evaluating the success of the Ready Up feature. Clear metrics helped mitigate concerns about potential misuse and provided a basis to demonstrate the feature's effectiveness.
Managing User Input Noise: Mikey highlighted the challenge of filtering out noise from extensive user input in product development. Utilizing measurable outputs and data-driven approaches helped Mikey stay focused on important aspects without compromising core values or altering the product's vision.
Incremental Product Adjustments: Mikey's approach involved making incremental tweaks to the game based on data-driven insights rather than implementing significant changes. Balancing the need for innovation with user satisfaction was crucial to avoid upsetting the user base.

Lukasz:

Differences in User Scale: It’s hard to not highlight the vast difference in the number of users and feedback between Mikey's gaming world and Zuora's business model. Despite fewer users, Zuora remains data-driven, but the dynamics of user feedback and testing strategies differ significantly from Mikey's context.
Context-Aware A/B Testing: I noted the necessity for context-aware A/B testing in Mikey's environment, where understanding the specific context is crucial for effective testing. Achieving a delicate balance in tweaking elements and minimizing negative consequences was crucial for long-standing success.
Monitoring User Feedback: I’m curious about Mikey's validation process for hypotheses and speculate that monitoring user feedback on message boards might be an organic validation method. Our conversation highlighted the challenge of ensuring that data-backed hypotheses align with the actual user experience and how user feedback plays a role in this validation process.

Segment Timestamps:

*(00:26) - Ship it or Skip it

*(06:38) - PM Power Moves

*(37:57) - Host Takeaways

Links: