Webinar: 99 Ways to Enrich Streaming Data with Apache Flink - Konstantin Knauf
Why Stack Overflow recommends it
Stack Overflow users link it when answering Flink questions about joining or enriching a stream with reference data, as an overview of the different approaches and their tradeoffs.
Recommended by 3 people across 5 different months, 2019 to 2022.
What it is
A one-hour Ververica webinar by Konstantin Knauf on enriching a fast data stream with slow-changing reference data in Apache Flink. He walks through the main approaches, from preloading data in a rich function to joins and broadcast streams, and covers the use cases, limits and tradeoffs of each.
What people said
Strong rec David Anderson · 2020-03 · Stack Overflow · on “How to join a stream and dataset?”
There are lots of ways to approach stream enrichment with Flink, depending on the exact requirements. [this video] is a good talk by Konstantin Knauf that covers many different approaches, and the tradeoffs between them. In the simple case where the enrichment data is immutable and reasonably small, I would just use a RichFlatMap and load the whole file in the open() method. That would look something like this: public class EnrichmentWithPreloading extends RichFlatMapFunction<Event, EnrichedEvent> { private Map<Long, SensorReferenceData> referenceData; @Override public void open(final Configuration parameters) throws Exception { super.open(parameters); referenceData = loadReferenceData(); } @Over
Recommends MIkCode · 2019-09 · Stack Overflow · on “Flink: Broadcast a map of values created using keyBy operator to another stream, cold start problem”
There is a several ways to implement what you are looking for , what is the TPS of the incoming strem? also check this video it might help you [this video]
Recommends David Anderson · 2020-09 · Stack Overflow · on “Apache Flink : How to Call One Stream from Another Stream”
There are many ways you might approach this. [this video] is an excellent introduction to the overall topic, and there are code examples here [LINK] .
Recommends snntrable · 2021-01 · Stack Overflow · on “What is the best way to have a cache of an external database in Flink?”
In essence David's answer summarizes it well. If you are looking for more detail: not long ago, I gave a webinar [1] on this topic including running code examples. [2] [1] [this video] [2] [LINK]
Recommends David Anderson · 2022-02 · Stack Overflow · on “In-Memory Lookup in Flink operators”
In addition to what @kkrugler has proposed, you could load the enrichment data in the open method of your enrichment function, and store it in a local variable (rather than in flink state). You could still periodically update it with a broadcast stream, but this would solve the state bootstrapping problem. (Since the ground truth of the state lives in a file, there's no benefit from using Flink state to store this data if you always read it from a file as the job starts.) One potential problem with pre-loading state in the open method is that if you want to have each instance only load the enrichment data for part of the total keyspace, there's no way to anticipate which keys each instance w