Multiprocessing

Bypass the GIL with separate processes, Process pools, shared memory, and inter-process communication.

Overview

The `multiprocessing` module spawns separate OS processes, each with its own Python interpreter and memory space. Because each process has its own GIL, they run truly in parallel on multiple CPU cores making multiprocessing the right tool for CPU-bound work. The trade-off: processes don't share memory, so communication requires explicit IPC (queues, pipes, shared memory). Spawning a process is also heavier than spawning a thread. **Pool** is the workhorse: `Pool(n)` creates `n` worker processes and distributes work via `.map()`, `.starmap()`, or `.apply_async()`. For finer control, `Process` gives you raw process management. `Queue` and `Pipe` handle inter-process communication. `Value` / `Array` provide shared memory with synchronisation. For data-science workloads, `concurrent.futures.ProcessPoolExecutor` provides a higher-level API that mirrors `ThreadPoolExecutor`.

Key Points

•`multiprocessing.Process(target=fn, args=(...))` raw process; `.start()` and `.join()`
•`Pool(n).map(fn, iterable)` distribute work across n processes and collect results
•`Pool.starmap(fn, [(a1,b1),(a2,b2)])` like map but unpacks tuple args
•`multiprocessing.Queue` thread- AND process-safe FIFO for IPC
•`multiprocessing.Pipe()` returns two Connection objects; fast for two-endpoint communication
•`Value` and `Array` shared memory with a `Lock` built in
•`concurrent.futures.ProcessPoolExecutor` modern, Executor-pattern interface
•Always guard entry point with `if __name__ == "__main__":` on Windows/macOS to prevent recursive spawning

Examples

Pool.map parallel CPU work

from multiprocessing import Pool
import time

def expensive(n):
    total = 0
    for i in range(n * 1_000_000):
        total += i
    return total

if __name__ == "__main__":
    nums = [5, 4, 3, 2]
    with Pool(4) as pool:
        results = pool.map(expensive, nums)
    print(results)

With 4 cores this completes in roughly the time of the longest single task, not the sum of all.

Inter-process communication with Queue

from multiprocessing import Process, Queue

def producer(q):
    for i in range(5):
        q.put(i * i)
    q.put(None)   # sentinel

def consumer(q):
    while True:
        val = q.get()
        if val is None:
            break
        print("got:", val)

if __name__ == "__main__":
    q = Queue()
    p1 = Process(target=producer, args=(q,))
    p2 = Process(target=consumer, args=(q,))
    p1.start(); p2.start()
    p1.join();  p2.join()

ProcessPoolExecutor with futures

from concurrent.futures import ProcessPoolExecutor, as_completed

def square(n):
    return n * n

if __name__ == "__main__":
    with ProcessPoolExecutor(max_workers=4) as ex:
        futures = {ex.submit(square, i): i for i in range(10)}
        for fut in as_completed(futures):
            n = futures[fut]
            print(f"square({n}) = {fut.result()}")

Common Mistakes

Omitting `if __name__ == "__main__":` on Windows causes infinite recursive spawning
Passing large objects through a Queue/Pipe they are pickled (serialised); prefer shared memory or file paths for big data
Using a Pool after it's closed always use the `with Pool() as pool:` context manager

Note

Use `concurrent.futures.ProcessPoolExecutor` for new code it's cleaner than raw Pool and integrates well with async code via `loop.run_in_executor`.

Multiprocessing

Overview

Key Points

Examples

Common Mistakes

Note

Exercises

Knowledge Check

Why does multiprocessing bypass the GIL?