Technology Deep Dive Track

Tuning Flink for Robustness and Performance

Flink's stateful stream processing engine presents a huge variety of optional features and configuration choices to the user. Figuring out the ""optimal"" choices for any production environment and use-case can therefore often be challenging. In this talk, we will explore and discuss the universe of Flink configuration with respect to robustness and performance.
We will start with a closer look under the hood, at core data structures and algorithms, to build the foundation for understanding the impact of tuning parameters and the costs-benefit-tradeoffs that come with certain features and options. In particular, we will focus on state backend choices (Heap vs RocksDB), tuning checkpointing (incremental checkpoints, ...) and recovery (local recovery), file systems, TTL state, and considerations for the network stack. This also includes a discussion about estimating memory requirements and memory partitioning.

Authors

Stefan Richter
data Artisans
Stefan Richter

Stefan is an Apache Flink comitter and works as a software engineer at data Artisans. He holds a PhD in Computer Science from Saarland University where he worked as researcher in the field of infomation systems. His research focus was on indexing, big data, and main memory databases.

Fill out the form to view
the Slides and Video

* All fields required