A Deeper Understanding of Spark Internals - Aaron Davidson (Databricks)

What people said

Recommends Francois G · 2015-01 · Stack Overflow · on “How does Spark read 100K images effeciently?”

I suspect that may be because you have a lot of small files on HDFS, which is a problem as such (the 'small files problem'). Here [LINK] you'll find a few suggestions in addressing the issue. You may also want to set the number of partitions (the minpartitions argument of wholetextFiles ) to a reasonable number : at least 2x the number of cores in your cluster (look there [this video] for details). But in sum, apart from the 2 ideas above, the way you're loading those is ok and not where your problem lies (assuming spark is your Spark context).

Recommends Saket · 2016-02 · Stack Overflow · on “Spark + Scala transformations, immutability & memory consumption overheads”

The memory requirements of Spark not 10 times if you have 10 transformations in your Spark job. When you specify the steps of transformations in a job Spark builds a DAG which will allow it to execute all the steps in the jobs. After that it breaks the job down into stages. A stage is a sequence of transformations which Spark can execute on dataset without shuffling. When an action is triggered on the RDD, Spark evaluates the DAG. It just applies all the transformations in a stage together until it hits the end of the stage, so it is unlikely for the memory pressure to be 10 time unless each transformation leads to a shuffle (in which case it is probably a badly written job). I would recomme

Recommends taotao.li · 2016-03 · Stack Overflow · on “Spark code organization and best practices”

I think you can subscribe Apache Spark , databricks channel on youtube, listen more and know more, especially for the experiences and lessons from others. Apache Spark [LINK] databricks [LINK] Spark Technology Center [LINK] here is some videos recommended: SparkUI Visualization [LINK] slide SparkUI Visualization [LINK] Spark in Production: Lessons from 100+ Production Users [LINK] slide Spark in Production: Lessons from 100+ Production Users [LINK] Spark Tuning for Enterprise System Administrators [LINK] Spark Tuning for Enterprise System Administrators [LINK] Building, Debugging, and Tuning Spark Machine Learning Pipelines - Joseph Bradley (Databricks) [LINK] slide Building, Debugging, and

Cites Sathish · 2015-06 · Stack Overflow · on “How DAG works under the covers in RDD?”

Even i have been looking in the web to learn about how spark computes the DAG from the RDD and subsequently executes the task. At high level, when any action is called on the RDD, Spark creates the DAG and submits it to the DAG scheduler. The DAG scheduler divides operators into stages of tasks. A stage is comprised of tasks based on partitions of the input data. The DAG scheduler pipelines operators together. For e.g. Many map operators can be scheduled in a single stage. The final result of a DAG scheduler is a set of stages. The Stages are passed on to the Task Scheduler.The task scheduler launches tasks via cluster manager (Spark Standalone/Yarn/Mesos). The task scheduler doesn't know ab

Cites Sathish · 2015-06 · Stack Overflow · on “How Spark works internally”

Spark revolves around the concept of a resilient distributed dataset (RDD), which is a fault-tolerant collection of elements that can be operated on in parallel. RDDs support two types of operations: transformations, which create a new dataset from an existing one, and actions, which return a value to the driver program after running a computation on the dataset. Spark translates the RDD transformations into something called DAG (Directed Acyclic Graph) and starts the execution, At high level, when any action is called on the RDD, Spark creates the DAG and submits to the DAG scheduler. The DAG scheduler divides operators into stages of tasks. A stage is comprised of tasks based on partitions

Cites Mateusz Dymczyk · 2016-03 · Stack Overflow · on “Performance vs readability in Spark -- Intermediate calculations”

The only overhead I can come up with is creating a val that points to an RDD on the driver. Not sure how many ms that takes Scala ;-) First of all in your example nothing would happen as you are not making Spark submit any jobs . A job is only submitted on actions (i.e. collect() etc.). The assignment you showed just declares an RDD but doesn't trigger any actions. Because of the above all RDDs can be lazy so you don't need to compute anything. When you submit a job Spark will create a directed acyclic graph (DAG) of the tasks (filter, map, reduce, groupByKey etc.) which have to be performed to get the job done :-) There are two types of tasks: narrow and wide . Wide tasks require a data shu