A look at Flink’s internal data structures & algorithms for efficient checkpointing - Stefan Richter
Why Stack Overflow recommends it
Stack Overflow users link it when explaining how Flink's state backends and checkpointing work internally, such as when state gets copied or serialized during asynchronous checkpoints.
Recommended by 1 people across 1 different months, 2019 to 2022.
What it is
A 28-minute conference talk from Flink Forward Berlin 2017 in which data Artisans engineer Stefan Richter explains how Apache Flink implements incremental and asynchronous checkpointing. He covers the underlying data structures, log-structured merge trees and copy-on-write hash tables, and how they fit together for fault tolerance at large state sizes.
What people said
Recommends David Anderson · 2019-05 · Stack Overflow · on “Apache Flink Dev documentation?”
I find that if you understand how some part of Flink works, the source code is generally understandable. The initial challenge then is to have a correct understanding of the expected behavior of the code. To that end, here are some helpful resources: The best starting point is Stream Processing with Apache Flink [LINK] by Fabian Hueske and Vasiliki Kalavri. Any significant development work done on Flink in recent years has been preceded by a Flink Improvement Proposal [LINK] . These are probably the best available resource for getting a deeper understanding of specific topics and areas of the code. The documentation [LINK] has a section on "Internals" that covers some topics. And there have
Cites Caesar · 2019-01 · Stack Overflow · on “Apache Flink: How often is state de/serialized?”
I think there is a little catch with the MemoryStateBackend and using de/serialization to create copies. [this video] explains it, but I forgot the details...
Cites Ben L. · 2022-06 · Stack Overflow · on “Apache Flink: How often is state de/serialized?”
I watched the video that Caesar mentioned, here is where they talk about it. [this video] . It will create a copy when getting it from heap state after a checkpoint while it's still asynchronously copying the data to the state file. This prevents the program from modifying the object while it's still being saved.