Watermarks: Time and Progress in Apache Beam and Beyond
Why Stack Overflow recommends it
Stack Overflow users link it when explaining Dataflow and Beam watermarks, data freshness and system latency metrics, since it is the in-depth explanation from a member of the Beam/Dataflow team.
Recommended by 3 people across 3 different months, 2019 to 2023.
What it is
A conference talk by Slava Chernyak of Google at Strata Hadoop NYC 2016 explaining watermarks, the mechanism streaming systems use to measure progress and completeness of out-of-order event-time data. It covers how the low watermark is defined and how it is applied in real Apache Beam and Dataflow pipelines.
What people said
Recommends Pablo · 2019-03 · Stack Overflow · on “Cloud Dataflow what is the exact definition of freshness and latency?”
These metrics are notoriously tricky. An in-depth dive into how they work can be seen in this talk by a member of the Beam / Dataflow team [this video] . Pipelines are split in series of computations that occur in memory, and computations that require serializing your data to some sort of data store. For example, consider the following pipeline: with Pipeline() as p: p | beam.ReadFromPubSub(...) \ | beam.Map(parse_data) | beam.Map(into_key_value_pairs) \ | beam.WindowInto(....) \ | beam.GroupByKey() \ | beam.Map(format_data) \ | beam.WriteToBigquery(...) This pipeline would get broken up into two stages . A stage is a series of computations that can be applied in memory. The first stage goes from
Recommends zangw · 2023-04 · Stack Overflow · on “Dataflow Watermark Concept”
Although robertwb [LINK] gives one wonderful answer [LINK] to this question. I want to summarize the basic concept of Watermark in Dataflow, based on Watermarks: Time and Progress in Apache Beam and Beyond [this video] , and hope help others to understand the watermark more clearly. Concept Watermarks are a mechanism for tracking the completeness of results in event time. The low watermark of a streaming step is the timestamp of the oldest work not yet completed. Usage Visibility: If the step is stuck, or failing, the low watermark will not advance. This lets us to detect and alert on stuck or failing steps. Actionability: A low watermark at time T guarantees that all work before T has been finis
Recommends Sakshi Gatyan · 2023-07 · Stack Overflow · on “DataFlow Data freshness metric definition”
Data freshness metric is basically a graph in Dataflow UI which shows the latency of your data in the pipeline. It very much correlates with system latency and is mainly is used as an indicator of performance and health of the pipeline. High spikes in the data freshness graph would indicate there is some delay (operations are slow) or any bottleneck in the pipeline. Which can increase the data backlog. Output watermark is the processing time of a tranformation stage or stages in a Dataflow pipeline. A detailed explanation can be found in this [this video] talk given by a member of Dataflow Eng team. One similar thread you can refer to is here [LINK] .
Cites Pablo · 2019-03 · Stack Overflow · on “How are Dataflow bundles created after GroupBy/Combine?”
There are two answers to your question. One is specific to your use case, and the other is in general about bundling / windowing in streaming. Specific to your pipeline I am assuming that the 'key' for Datastore is the User ID? In that case, if you have events from the same user in more than one window, your GroupByKey or Combine operations will have one separate element for every pair of user+window. So the question is: What are you trying to insert into datastore? An individual user's resulting aggregate over all time ? In that case, you'd need to use a Global Window. A user's resulting aggregate for every 30 seconds in time? Then you need to use the window as part of the key you use to in
Cites Pablo · 2019-09 · Stack Overflow · on “How can I debug why my Dataflow job is stuck?”
The first resource that you should check is Dataflow documentation. It should be useful to check these: Troubleshooting your Pipeline [LINK] Common error guidance [LINK] If these resources don't help, I'll try to summarize some reasons why your job may be stuck, and how you can debug it. I'll separate these issues depending on which part of the system is causing the trouble. Your job may be: Job stuck at startup A job can get stuck being received by the Dataflow service, or starting up new Dataflow workers. Some risk factors for this are: Did you add a custom setup.py file? Do you have any dependencies that require a special setup on worker startup? Are you manipulating the worker container?
Cites allen · 2020-03 · Stack Overflow · on “Is apache-beam a good choice when the event time ordering has to preserved when writing to sink?”
The other option while using apache beam is to have parallel writes and make sure the database is usable only till the output watermark time of the last beam stage(requires database support) [this video]