Leon Barrett - Clojure Parallelism Beyond Futures
Why Stack Overflow recommends it
Stack Overflow users link it when answering questions about pmap behavior, laziness, CPU saturation and managing threads in Clojure.
Recommended by 3 people across 3 different months, 2016 to 2018.
What it is
A 30-minute Clojure/West conference talk by Leon Barrett, author of the Claypoole library, on how parallelism works in Clojure. It explains how future and pmap work internally, covers their limitations, and introduces alternatives such as reducers, tesser and claypoole.
What people said
Recommends Ben Kovitz · 2016-05 · Stack Overflow · on “Why is pmap|reducers/map not using all cpu cores?”
This talk [this video] by Leon Barrett, author of Claypoole [LINK] , might have some relevant information. It explains `pmap` in detail, including why it often doesn't saturate the CPU, and a little bit about why feeding one `pmap` into another can have surprising results. Also, if you're mainly looking for a way to saturate your CPU, Claypoole might be exactly what you need.
Recommends Alan Thompson · 2017-05 · Stack Overflow · on “Clojure: Create and manage multiple threads”
It is hard to be precise w/o knowing the details of your problem. There are several choices as you mention: Plain Java threads & threadpools. If your problem is similar to a pre-existing Java solution, this may be the most straightforward. Simple Clojure threading with future et al. Kicking off a thread with future and getting the result in a promise is very easy. Replace map with pmap (parallel map). This can help in simple cases that are primarily map/reduce oriented. The Claypoole library: Lots of tools to make multithreading simpler and easier. Please see their GitHub project [LINK] and the Clojure/West talk. [this video]
Recommends Leon Barrett · 2018-03 · Stack Overflow · on “can you explain pmap laziness and memory footprint?”
While Taylor's answer is correct, I also gave a presentation on what happens inside of pmap , and how it's lazy, at Clojure West a few years ago. I know not everyone likes videos for learning, but if you do, it might be helpful: [this video] (If you want non-lazy pmap , I second the endorsement for Claypoole.)