Solving for Timeseries with Apache Beam (Beam Summit Europe 2019)
Why Stack Overflow recommends it
Stack Overflow users link it when answering Apache Beam questions about joining or carrying forward time series values, since it shows how state and timers handle missing data and validity windows.
Recommended by 3 people across 3 different months, 2019 to 2020.
What it is
A Beam Summit Europe 2019 session by Google's Reza Rokni and Tyler Akidau on processing time series in Apache Beam. It covers ordered processing across window boundaries using Global Windows, State and the Timer API, and introduces the idea of Validity Windows.
What people said
Recommends Guillem Xercavins · 2019-09 · Stack Overflow · on “Merge two PCollection (Apache beam)”
To help with 'validity windows' you can look into the State and Timers APIs. I recommend this blog post [LINK] and talk [this video] about the topic.
Recommends Kenn Knowles · 2020-08 · Stack Overflow · on “Merge two PCollection (Apache beam)”
The comment by @jszule is a good answer in general for Dataflow/Beam: The best supported join is when the two PCollections have a common key. For most data Beam can figure out a schema and you can use CoGroup.join [LINK] transform. The design decision you have to make is how to choose the keys, such as rounding down to the nearest 1000. Your use case has a complication: you need to carry forward values in a time series for keys that have no data. The solution is to use state and timers to generate the "missing" values. You will still need to carefully choose keys since state and timers are per-key-and-window. State and timers also work in batch mode so this is a batch/streaming unified solut
Recommends Reza Rokni · 2020-10 · Stack Overflow · on “Apache Beam - Streaming Join on Temporal Relation”
You might find this interesting as well, things have progressed since then but concepts are still valid: [this video] Also a WIP but might be of interest: [LINK]