Technology Deep Dive Track

Elastic Streams at Scale

One of the big operational challenges when running streaming applications is to cope with varying workloads. Variations, e.g. daily cycles, seasonal spikes or sudden events, require that allocated resources are constantly adapted. Otherwise, service quality deteriorates or money is wasted. Apache Flink 1.5 includes a lot of enhancements to support full resource elasticity on cluster management frameworks such as Apache Mesos. With the latest version, it is now possible to build elastic applications which can be programmatically scaled up or down in order to react to changing workloads. In this talk, we will discuss recent improvements to Flink's deployment model which also enables full resource elasticity. In particular, we will discuss how Flink leverages cluster management frameworks, e.g. Mesos, and already-introduced features like scalable state to support elastic streaming applications. We will conclude the presentation with a short demo showing how a stateful Flink application can be rescaled on top of Mesos.

Authors

Till Rohrmann
data Artisans
Till Rohrmann

Till is a PMC member of Apache Flink and software engineer at dataArtisans. His main work focuses on enhancing Flink’s scalability as a distributed system. Till studied computer science at TU Berlin, TU Munich and École Polytechnique where he specialized in machine learning and massively parallel dataflow systems.

Joerg Schad
Mesosphere
Joerg Schad

Jörg is the technical lead for community projects at Mesosphere in San Francisco. In his previous life he implemented distributed and in memory databases and conducted research in the Hadoop and Cloud area during his PhD. His speaking experience includes various Meetups, international conferences, and lecture halls.

Fill out the form to view
the Slides and Video

* All fields required