streaming data

This episode delves into Apache Flink, a versatile platform for executing both batch and real-time streaming data analysis tasks. This session marks the beginning of a three-part series unveiling Amazon Web Services' (AWS) new managed service built on Flink. Future episodes will explore this service in detail and examine customer experiences.

The podcast features insights from Danny Cranmer, a principal engineer at AWS and an Apache Flink PMC and Committer, along with Hong Teoh, a software development engineer at AWS.

Flink stands out as a high-level framework for defining data analytics jobs, accommodating both batch and streaming data sets. It offers APIs for building analysis jobs in various languages, including Java, Python, and SQL. Flink also provides a distributed job execution engine with fault tolerance and horizontal scaling capabilities.

One prominent use case is Extract-Transform-Load (ETL), where raw data is swiftly processed for specific workloads. Flink excels in delivering low-latency transformations for unbounded data streams. Additionally, Flink supports event-driven applications, responding immediately to triggers such as user requests for weather data.

Flink ensures exactly-once processing, critical for scenarios like financial transactions. It employs checkpoints to maintain data integrity in case of node failures.

The podcast also touches on AWS's role in supporting the open-source Flink project and the future outlook for this powerful data processing framework.

Learn more from The New Stack about Apache Flink:

3 Reasons Why You Need Apache Flink for Stream Processing

Apache Flink for Unbounded Data Streams

8 Real-Time Data Best Practices

Kyle Weller (@KyleJWeller, Head of Product @onehousehq) talks about the latest trends in  OSS Data Lakes, Data Warehouses, and the evolution to “Data Lakehouses” with Apache Hudi

SHOW: 694

CLOUD NEWS OF THE WEEK - http://bit.ly/cloudcast-cnotw

NEW TO CLOUD? CHECK OUT - "CLOUDCAST BASICS"

SHOW SPONSORS:

• Datadog Synthetic Monitoring: Frontend and Backend Modern Monitoring
• Ensure frontend issues don’t impair user experience by detecting user-facing issues with API and browser tests with a free 14 day Datadog trial. Listeners of The Cloudcast will also receive a free Datadog T-shirt. 
• Solve your IAM mess with Strata's Identity Orchestration platform
• Have an identity challenge you thought was too big, too complicated, or too expensive to fix? Let us solve it for you! Visit strata.io/cloudcast to share your toughest IAM challenge and receive a set of AirPods Pro
• How to Fix the Internet (A new podcast from the EFF)

SHOW NOTES:

• Onehouse (homepage)
• Onehouse raises $25M Series A funding
• Apache Hudi (homepage)
• Delta Lake (homepage)
• Apache Iceberg (homepage)
• ​​Apache Hudi vs Delta Lake vs Apache Iceberg - Lakehouse Feature Comparison

Topic 1 - Welcome to the show. Tell us a little bit of your background, and where you focus your efforts at Onehouse?

Topic 2 - Your focus is on an emerging open source project, Apache Hudi. Before we dive into the project and technologies, we’re always interested in the background of what drove the creation of new projects. What problems existed before Hudi? 

Topic 3 - Let’s dive into Hudi. Data lakes, Delta Lakes, Lake houses, Icebergs. What is going on with all these water metaphors?  

Topic 4 - Hudi is focused on streaming data lakes. What are some of the things (types of applications) that need a streaming data lake? Where do transactions come into play? Where do data warehouse capabilities come into play?

Topic 5 - Stitching together open source projects and platforms can be complicated. How does the Onehouse platform simplify all of this for either data scientists or platform teams?

Topic 6 - What are some examples of how companies are using Onehouse and Hudi today? 

FEEDBACK?

• Email: show at the cloudcast dot net
• Twitter: @thecloudcastnet

In this episode we talk about all things Bayesian. What is Bayesian inference and why is it the cornerstone of Data Science?

Bayesian statistics embodies the Data Scientist and their role in the data modelling process. A Data Scientist starts with an idea of how to capture a particular phenomena in a mathematical model - maybe derived from talking to experts in the company. This represents the prior belief about the model. Then the model consumes data around the problem - historical data, real-time data, it doesn't matter. This data is used to update the model and the result is called the posterior.

Why is this Data Science? Because models that react to data and refine their representation of the world in response to the data they see are what the Data Scientist is all about.

We talk with Dr Joseph Walmswell, Principal Data Scientist at life sciences company Abcam, about his experience with Bayesian modelling.

Further Reading

• Publication list for Dr. Joseph Walmswell (https://bit.ly/3s8xluH via researchgate.net)
• Blog on Bayesian Inference for parameter estimation (https://bit.ly/2OX46fV via towardsdatascience.com)
• Book Chapter on Bayesian Inference (https://bit.ly/2Pi9Ct9 via cmu.edu)
• Article on The Monty Hall problem (https://bit.ly/3f1pefr via Wikipedia)
• Podcast on  "The truth about obesity and Covid-19", More or Less: Behind the Stats podcast (https://bbc.in/3lBqCGS via bbc.co.uk)
• Gov.uk guidance:
  • Article on "Understanding lateral flow antigen testing for people without symptoms" (https://bit.ly/313JDs9)
  • Article on "Households and bubbles of pupils, students and staff of schools, nurseries and colleges: get rapid lateral flow tests" (https://bit.ly/3c5ZXih)

Some links above may require payment or login. We are not endorsing them or receiving any payment for mentioning them. They are provided as is. Often free versions of papers are available and we would encourage you to investigate.

Recording date: 16 March 2021
Interview date: 26 February 2021

Thanks for joining us in the DataCafé. You can follow us on twitter @DataCafePodcast and feel free to contact us about anything you've heard here or think would be an interesting topic in the future.

• Teralytic Homepage
• Teralytic Blog
• Teralytic Sensors Help Farmers Manage Their Fertilizing
• [PODCAST] @PodCTL - Containers | Kubernetes -  RSS Feed, iTunes, Google Play, Stitcher, TuneIn and all your favorite podcast players
• [A CLOUD GURU] Get The Cloudcast Alexa Skill
• [A CLOUD GURU] A Cloud Guru Membership - Start your free trial. Unlimited access to the best cloud training and new series to keep you up-to-date on all things AWS.
• [A CLOUD GURU] FREE access to AWS Certification Exam Prep Guide - At A Cloud Guru, the #1 question received from students is "I want to pass the AWS cert exam, so where do I start?" This course is your answer.
• [FREE] eBook from O'Reilly

• Email: show at thecloudcast dot net
• Twitter: @thecloudcastnet and @ServerlessCast

The aviation industry is grappling with another tragedy, the crash of a Germanwings Airbus A320 in the French Alps. Audio from the cockpit voice recorder indicates that the captain was locked out of the cockpit, and couldn’t get back in, though the flight data recorder has yet to be found. This crash – and the fact that sourcing information from the physical black boxes is crucial to understanding what happened in any accident - has reignited the conversation over whether aircraft should stream some level of black box data in real-time, and whether there should be video cameras in the cockpit. Co-hosts Max Flight and Mary Kirby talk about why the chorus of voices calling for change is growing louder, and explain why black box streaming is relative to #PaxEx.

Next, Neelam has been writing a series of articles for Runway Girl Network about India’s efforts to attract women to aviation. She tells us why SpiceJet and other carriers are in a hiring push, and how they're getting the word out via social media. She notes that women from all walks of life - not just the "elite" daughters of Indian pilots - are increasingly drawn to this profession in her country.

Last but not least, Neelam draws on her deep knowledge of the Indian aviation scene to bring us up to speed on how Indian manufacturers are growing their footprint in the aerospace supply chain in partnership with Airbus and Boeing. We also consider the opportunities for Indian firms to ultimately break into the highly regulated #PaxEx market.