dataflow

Classic tool suites that are used to process network flow records deal with very limited detail on the network connections they summarize. These tools limit detail for several reasons: (1) to maintain long-baseline data, (2) to focus on security-indicative data fields, and (3) to support data collection across large or complex infrastructures. However, a consequence of this limited detail is that analysis results based on this data provide information about indications of behavior rather than information that accurately identifies behavior with high confidence. In this webcast, Tim Shimeall and Katherine Prevost discuss how to use IPFIX-formatted data with detail derived from deep packet inspection (DPI) to provide increased confidence in identifying behavior. 

Jeff Meyerson talks with Frances Perry about Apache Beam, a unified batch and stream processing model. Topics include a history of batch and stream processing, from MapReduce to the Lambda Architecture to the more recent Dataflow model, originally defined in a Google paper. Dataflow overcomes the problem of event time skew by using watermarks and other methods discussed between Jeff and Frances. Apache Beam defines a way for users to define their pipelines in a way that is agnostic of the underlying execution engine, similar to how SQL provides a unified language for databases. This seeks to solve the churn and repeated work that has occurred in the rapidly evolving stream processing ecosystem.