Thinking about Concurrency, Raymond Hettinger, Python core developer
Why Stack Overflow recommends it
Stack Overflow users link it when answering questions about Python threading, multiprocessing, the GIL and deadlocks, as an intro to choosing and structuring concurrency.
Recommended by 9 people across 9 different months, 2016 to 2021.
What it is
A conference keynote by Python core developer Raymond Hettinger on how to reason about concurrency in Python. He compares threads, processes and async, and argues for structuring programs around atomic message queues instead of shared state and locks.
What people said
Strong rec Efron Licht · 2017-04 · Stack Overflow · on “Is possible to keep spacy in memory to reduce the load time?”
Your fundamental problem here is launching a new script for every request. Instead of running a script for every request, run a function from within the script on every request. There are a variety of ways to handle user requests. The simplest is to periodically poll for requests and add them to a queue. The async framework is also useful for this kind of work. This talk by raymond hettinger [this video] is an excellent introduction to concurrency in Python.
Strong rec Good Morning Captain · 2017-05 · Stack Overflow · on “Multiple threads that communicate with each other”
Raymond Hettinger gave a really good keynote talk about the proper way to think about concurrency and multithreading here: [this video] And his notes can be found here: [LINK] What I recommend, which is from the talk, is to use an atomic message queue to "talk" between the threads. However, this talk and Raymond's work is done in 3.5 or 3.6. This library [LINK] will help you significantly.
Strong rec user48956 · 2017-08 · Stack Overflow · on “python locking and threading concurrency”
Its not true that threading has no upside. Use threading when you have a large amount of state to share between tasks. Use multiprocessing where you do not and the overheads for forking are high (e.g. when you need to share many kb of state between the processes). Here's a great video by a core python contributor [this video] You *DO* need to be careful to avoid forking after creating a thread. I think number crunching usually happens outside the GIL (C libraries should typically release the GIL while doing their processing).
Recommends wwii · 2016-11 · Stack Overflow · on “Multi-processing on Large Image Dataset in Python”
I just watched and practiced Thinking about Concurrency, Raymond Hettinger [this video] - it is worthwhile and should get you started.
Recommends Vivek Joshy · 2017-06 · Stack Overflow · on “Is multithreading in python a myth?”
No, Python does have multithreading. In fact, it uses system threads. The problem is just that it can't use more than one of the available cores. This is due to something called the GIL(Global Interpreter Lock). [LINK] Python threads still work for I/O bound tasks as opposed to CPU bound tasks which may cause deadlocks and race conditions. Many Python libraries solve this issue by using C extensions to bypass the GIL. Of course, this is all in the case of CPython. There is a very interesting talk about this by one of the core developers of Python. Thinking about concurrency, Raymond Hettinger [this video] Now you are right, it is much better to use multiprocessing to get the benefit of all the co
Recommends James Schinner · 2018-01 · Stack Overflow · on “Running multiple functions that make HTTP requests in parallel”
I can't give that question justice. The answer is moderately nuanced. This talk by Raymond Hettinger might be a good start: [this video]
Recommends jpmc26 · 2019-07 · Stack Overflow · on “How can I run the exact same function several times with different inputs and do so simultaniously”
Multithreading won't work unless the bottleneck is I/O anyway. Does each call need to read a lot of data from an external sources (files on disk, a database, a website)? This intro to concurrency [this video] would be a good place to start for you.
Recommends EvensF · 2021-05 · Stack Overflow · on “How to do two tasks at the same time in python”
To understand why the problem you are describing is more complicated than you might think, have a look at this talk of Raymond Hettiger about Concurrency ( [this video] )
Recommends gdahlm · 2021-07 · Stack Overflow · on “How do I avoid deadlock when using subprocess.Popen to connect multiple processes by pipes?”
It depends on what you are trying to accomplish as preferred style of development and reasons of trying to add concurrency will change the answer. I would be tempted to just use run() in series first unless you have a reason to add complexity. The issue with deadlocks and concurrency, especially when using the lower level Popen() are far more complex than I can answer with a few 100 chars. Here is a video that will help understand the pitfalls [this video] Personally, I would use _async for loops_ and because deadlocks are hard and the task is not truly parallel.
Cites joydeep bhattacharjee · 2018-05 · Stack Overflow · on “multi-threading with a common function in python”
As @lutz had written, there is no danger of race conditions in case there are no shared data between two instances of the common_function . This is what is meant by referential transparency and generally all programming languages aim that such functions should be thread safe . Sometimes, you will need to write and use functions which change some global state. In such cases the modern adage is to use event driven programming - which means to not communicate directly between threads but communicate via some thread safe queueing system. In python I am a huge fan of the queue module [LINK] for that matter. Another good queue module is the multiprocessing.queue for which a good example is here [L