ask confluent

It’s the first work-from-home episode of Ask Confluent, where Gwen Shapira (Core Kafka Engineering Leader, Confluent) virtually sits down with Apache Kafka® expert Anna McDonald (Staff Technical Account Manager, Confluent) to answer questions from Twitter. 

Find out Anna’s favorite Kafka Improvement Proposal (KIP), which  will start to use racially neutral terms in the Kafka community and in our code base, as well as answers to the following questions: 

• If you could pick any one KIP from the backlog that hasn't yet been implemented and have it immediately available, which one would you pick?
• Are we able to arrive at any formula for identifying the consumer/producer throughput rate in Kafka with the given hardware specifications (CPU, RAM, network, and disk)? 
• Does incremental cooperative rebalancing also work for general Kafka consumers in addition to Kafka Connect rebalancing?

They also answer how to determine throughput and achieve your desired SLA by using partitions. 

EPISODE LINKS

• Watch Ask Confluent #18: The Toughest Questions ft. Anna McDonald
• From Eager to Smarter in Apache Kafka Consumer Rebalances
• Streaming Heterogeneous Databases with Kafka Connect – The Easy Way
• Keynote: Tim Berglund, Confluent | Closing Keynote Presentation | Kafka Summit 2020
• Join the Confluent Community Slack
• Learn more with Kafka tutorials, resources, and guides at Confluent Developer
• Live demo: Kafka streaming in 10 minutes on Confluent Cloud
• Use 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)

Ask Confluent is back! From questions on Apache Kafka®, data integration, and log aggregation, to potential interview questions that Tim would ask if he were to interview himself, anything goes. 

If you're already a Kafka expert (or any type of expert), think about becoming a speaker. Gwen and Tim talk through how to submit a proposal and get accepted to conferences. As experienced conference goers, they explain that what makes a successful talk is making sure you present for the attendee instead of making it about yourself. In essence, what can your idea or code do to help someone else? 

From there, the pair chat about the secret for a long marriage, REST Proxy and where it exists in Confluent Operator, how Kafka relates to Splunk when aggregating logs, and whether Tim can start making some use case based video content so that people can better understand Kafka and how it works. For those who have just started integrating Kafka, Tim and Gwen also provide some pointers about how to go about understanding it. 

EPISODE LINKS

• Confluent REST Proxy Documentation
• What is Apache Kafka?
• Splunk Connect for Kafka – Connecting Apache Kafka with Splunk
• Watch the video version of this podcast
• Join the Confluent Community Slack

Vinoth Chandar has led various infrastructure projects at Uber and is one of the main drivers behind the ksqlDB project. In this episode hosted by Gwen Shapira (Engineering Manager, Cloud-Native Apache Kafka®), Vinoth and Gwen discuss what ksqlDB is, the kinds of applications that you can build with it, vulnerabilities, and various ksqlDB use cases. They also talk about what's currently the best version of Apache Kafka version for performance improvements that don’t cause breaking changes to existing Kafka configuration and functionality. 

EPISODE LINKS

• Read about ksqlDB on the blog
• Learn more about ksqlDB
• ksqlDB Demo | The Event Streaming Database in Action
• Follow ksqlDB on Twitter
• What’s New in Apache Kafka 2.3
• What is Apache Kafka? 
• Watch the video version of this podcast
• Join the Confluent Community Slack
• Fully managed Apache Kafka as a service! Try free.

Gwen Shapira (Core Kafka Software Engineer, Confluent) sits down to answer the questions you've had about event streaming, Apache Kafka®, Confluent, and everything in between. This includes creating tables in nested JSON topics, how to balance ordering, latency and reliability, building event-based systems, and how to navigate the tricky endOffsets API. She talks about the hardships of fencing Zombie requests, some of the talks given at previous Kafka Summits, and an important question from Ask Confluent #3. 

EPISODE LINKS

• KIP-91: Provide Intuitive User Timeouts in The Producer
• KIP-79: ListOffsetRequest/ListOffsetResponse v1 and add timestamp search methods to the new consumer
• KSQL recipe on creating tables in a nested JSON topic
• Data Wrangling with Apache Kafka and KSQL
• Struct (Nested Data) | Level Up Your KSQL
• Kafka Summit 2018 Keynote (Experimentation Using Event-Based Systems)
• Apache Kafka 2.3
• endOffsets documentation
• Jun Rao, Confluent - Kafka Controller: A Deep Dive
• Confluent Platform 5.3
• Kafka Summit 2017 Keynote (Go Against the Flow: Databases and Stream Processing)
• The Event Streaming Platform Explained
• Watch the video version of this podcast
• Join the Confluent Community Slack

Is Apache Kafka® actually a database? Can you install Confluent Control Center on Google Cloud Platform (GCP)? All this, plus some tips from Dan Norwood, the first user of Kafka Streams.

EPISODE LINKS

• Control Center Docker image
• Control Center Docker configuration
• Complete Streams example
• Watch the video version of this podcast
• Join us in Confluent Community Slack

Gwen and Kai chat about machine learning architectures, and whether software engineers and data scientists can learn to get along.

EPISODE LINKS

• Blogs on deploying machine learning workloads: 
  • Machine Learning with Python, Jupyter, KSQL and TensorFlow
  • How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka
  • Using Apache Kafka to Drive Cutting-Edge Machine Learning
• KIP-392: Allow consumers to fetch from closest replica 
• Watch the video version of this podcast

Stanislav Kozlovski joins us to discuss common pitfalls when using Kafka consumers and a new KIP that promises to make consumer restarts much smoother.

EPISODE LINKS

• KIP-345: Static consumer membership 
• KIP-211: Documents the current behavior of offset expiration
• Watch the video version of this podcast
  
  

Do metrics for detecting clients from old versions actually exist? Or is Gwen making features up? This and more useful advice is coming up on today's episode of Ask Confluent.

EPISODE LINKS

• The Java property that will refresh DNS cache frequently: java.security.Security.setProperty(“networkaddress.cache.ttl” , “60");
• Improvements to DNS lookups in Confluent Platform 5.1.2 (Apache Kafka 2.1.1):
  • KAFKA-7755
  • KAFKA-7890
• More reasons to upgrade to Confluent Platform 5.1.2
• Monitoring clients with old versions:
  • KIP-188 has lots of important new metrics 
• If you are worried about “down-conversion” as discussed in Ask Confluent #5, you want to monitor: MBean: kafka.server:type=BrokerTopicMetrics,name=FetchMessageConversionsPerSec,topic=([-.\w]+)
• KIP-188 also added a metric for temp memory usage (memory used for conversion and compression) that can be useful
• In KIP-272, we’ve added version tag to request metrics, so you can see how many requests per sec you get from each version
• Recommendations for Kafka Summit NYC
• Watch the video version of this podcast

Want to know how Kafka Connect distributes tasks to workers? Always thought Connect rebalances could be improved? In this episode of Ask Confluent, Gwen Shapira speaks with Konstantine Karantasis, software engineer at Confluent, about the latest improvements to Kafka Connect and how to run the Confluent CLI on Windows.

EPISODE LINKS

• Improved rebalancing for Kafka Connect
• Improved rebalancing for Kafka Streams
• The "what would Kafka do?" scenario from Mark Papadakis
• The future of retail at Nordstrom
• Watch the video version of this podcast 

Gwen asks: What happens when garbage collection causes Kafka to pause? And how do we run a Schema Registry cluster? We’ll find out in this episode of Ask Confluent.

In "Ask Confluent," Gwen Shapira (Software Engineer, Confluent) and guests respond to a handful of questions and comments from Twitter, YouTube and elsewhere.

EPISODE LINKS

• ZooKeeper connection timeout configuration: zookeeper.connection.timeout.ms, as we said, this defaults to 6,000
• Schema Registry failover instructions
• Watch the video version of this podcast

Gwen is joined in studio by special guest Guozhang Wang, Kafka Streams pioneer and engineering lead at Confluent. He’ll talk to us about standby tasks and how one deserializes message headers. In "Ask Confluent," Gwen Shapira (Data Architect, Confluent) and guests respond to a handful of questions and comments from Twitter, YouTube and elsewhere.

EPISODE LINKS

• Documentation of standby tasks, including configs
• Events with different schema in same topic
• How to populate a database from Kafka and solve the parent-child relation problem
• Watch the video version of this podcast

Gwen is joined in studio by special guest Matthias J. Sax, a software engineer at Confluent. He’ll talk to us about Kafka consumers and Kafka Streams failover. In "Ask Confluent," Gwen Shapira (Data Architect, Confluent) and guests respond to a handful of questions and comments from Twitter, YouTube and elsewhere.

EPISODE LINKS

• Watch the video version of this podcast

Gwen is joined in studio by special guest Jason Gustafson, a Kafka PMC member and engineer at Confluent. He’ll talk to us about the big questions on Kafka architecture— number of partitions and exactly once. In "Ask Confluent," Gwen Shapira (Data Architect, Confluent) and guests respond to a handful of questions and comments from Twitter, YouTube and elsewhere.

EPISODE LINKS

• Hardening Kafka Replication
• Kafka open issues
• Watch the video version of this podcast

Gwen is joined in studio by co-host Tim Berglund and special guest, Viktor Gamov, a new member of Confluent’s Developer Experience Team specializing in Kafka, KSQL and Kubernetes. In this episode, we’ll find out: Does Viktor know what he’s talking about?

EPISODE LINKS

• Watch the video version of this podcast

Want to see a feature implemented in KSQL or other Kafka-related project? Gwen answers your questions from YouTube and walks through how to use GitHub issues to request features. This is the episode #4 of "Ask Confluent," a segment in which Gwen Shapira and guests respond to a handful of questions and comments from Twitter, YouTube and elsewhere.

EPISODE LINKS

• Watch the video version of this podcast

Tim Berglund and Gwen Shapira have a discussion with Koelli Mungee (Customer Operations Lead, Confluent) and cover the latest Apache Kafka upgrades, cloud APIs, and data durability. This is episode #3 of "Ask Confluent," a segment in which Gwen Shapira and guests respond to a handful of questions and comments from Twitter, YouTube, and elsewhere.

EPISODE LINKS

• Watch the video version of this podcast
• Fully managed Apache Kafka as a service! Try free.

Gwen Shapira answers your questions and interviews Sam Hecht (Head of Support, Confluent). This is the second episode of "Ask Confluent," a segment in which Gwen Shapira and guests respond to a handful of questions and comments from Twitter, YouTube and elsewhere.

EPISODE LINKS

• Watch the video version of this podcast

Tim Berglund and Gwen Shapira discuss Kubernetes, Confluent Operator, Kafka, KSQL, and more. This is the first episode of "Ask Confluent," a segment in which Gwen Shapira and guests respond to a handful of questions and comments from Twitter, YouTube and elsewhere.

EPISODE LINKS

• Watch the video version of this podcast