Cassandra and Spark Optimizing for Data Locality - Russell Spitzer (DataStax)
Why Stack Overflow recommends it
Stack Overflow users link it when explaining how the Spark-Cassandra connector reads data locally and splits token ranges across executors.
Recommended by 3 people across 3 different months, 2016 to 2021.
What it is
A 15-minute Spark Summit 2015 talk by Russell Spitzer of DataStax on how the Spark Cassandra connector uses data locality. It covers how Spark executors are matched to Cassandra token ranges so work is done on local nodes.
What people said
Strong rec Artem Aliev · 2018-01 · Stack Overflow · on “How Spark writes/reads process through spark-Cassandra connector different from CQLSH read/write process”
Here is a good explanation. I also recommend other Russell talks, if you want to understand spark-cassandra-connector internals Cassandra and Spark Optimizing for Data Locality - Russell Spitzer (DataStax) [this video]
Recommends Stefan Podkowinski · 2016-04 · Stack Overflow · on “Spark with replicated Cassandra nodes”
Yes it does. The Spark-Cassandra connector is data locality aware, i.e. each Spark node co-located with a Cassandra node will make sure to only process the local Cassandra data, which avoids shuffling lots of data across the network. You can find out how this works by watching a talk by Russell Spitzer on this topic here [this video] .
Recommends Nikola Yovchev · 2021-07 · Stack Overflow · on “How Spark-cassandra-connector determines the range to query on Cassandra?”
Spark-cassandra-connector basics The spark-cassandra-connector has fairly complicated internals, but the most important things (overly simplified) are the following: the connector would naturally prefer to query locally . E.g to avoid network and to have the spark executor query its local cassandra node to do that, the driver needs to understand the Cassandra topology and where the token ranges you need to query are (there is an initial ring describe done by the driver, so after that there is a full understanding where to find what part of your token) after understanding where the token ranges are, and mapping each token to an IP, the connector spreads the work in such a way that each local