Managing continuous applications Track

Extending Flink metrics: Real-time BI atop existing Flink streaming pipelines

Flink metrics module allows to use Dropwizard-like metrics and reporters in Flink pipelines. It opens a rich opportunity to not only monitor health of Flink pipelines but also attach real-time business intelligence metrics to run alongside existing Flink data jobs, thus avoiding a need to build a separate BI data-flow infrastructure. However, most of the work still resides with application developers: they have to define how those metrics are registered, calculated and attached to Flink functional operators – and then Flink will do the computation and reporting. The most challenging task is to add BI-like metrics, with key dimensions dynamically extracted from streamed data and metric values likely needed to be aggregated over multiple Flink tasks. We present a toolkit that extends Flink metrics and allows to decorate existing Flink operators with both simple health-check and complex BI-like metrics, updated and observable in real-time through IT monitoring and BI visualization dashboards. This work is based on a real-world production use-case of computing retail merchandise prices in real-time for the Walmart e-commerce catalog using Flink.

Authors

Andrew Torson
Principal Data Engineer Walmart Labs
Andrew Torson

Andrew Torson is a Principal Data Engineer in the Smart Pricing team within the Walmart Labs organization. His current work is focused on big-fast-data pipelines in Flink and Spark for the Walmart e-commerce business, leveraging ML-based retail pricing algorithms. Before joining Walmart Labs, Andrew worked as a data scientist and data engineer on a handful of IoT projects in the area of mobile robotics (warehousing/manufacturing/marine container terminals industries), using ML-based tools in Scala, Python and Java. Andrew holds a PhD in Operations Management from NYU and also worked as a ML scientist in the Siemens Research/Labs after his graduation, which led him towards his current product data engineer track.

Fill out the form to view
the Slides and Video

* All fields required