multiprocessing

John Melesky (PDX Python, June 2010)

In the 2.6 standard library

Otherwise, available from PyPI (>=2.4)

A note

multiprocessing is a huge library, and deserving of a full-fledged talk (or several), rather than a module-of-the-month.

The manager functionality, for example, is elaborate and useful.

A note

multiprocessing is a huge library, and deserving of a full-fledged talk (or several), rather than a module-of-the-month.

The manager functionality, for example, is elaborate and useful (as far as i can tell. i haven't used it. this is still just a motm talk).

A different note

multiprocessing is complex, and occasionally slightly hackish. Expect the following:

• These examples may not work in the REPL
• Similar arguments may have different semantics depending on whether they're passed to Process or Pool
• It "works" on Windows
• Overall, YMMV

Pre-multiprocessing

• fork/exec
• threading

Pre-multiprocessing

• fork/exec
• threading (inspired by Java!)

Process

sample process code

Process

• name, pid, exitcode, authkey
• start()
• run()
• join([timeout])
• is_alive()
• terminate()
• daemon

Pool

sample pool code

Pool

• apply(func[, args[, kwds]])
• apply_async(ditto, plus callback)
• map, map_async, imap, imap_unordered
• close()
• terminate()
• join()

Queue

(nonworking) sample queue code

Queue

See Queue.Queue

• empty(), full(), qsize()
• put(item[, block[, timeout]])
• put_nowait(item)
• get([block[, timeout]])
• get_nowait()

Pipe

(semiworking) sample pipe code

Connection

• send(obj)
• recv()
• fileno()
• close()
• poll([timeout])
• send_bytes, recv_bytes, recv_bytes_into

Stuff i'm not covering

• Lock
• shared memory maps (Value, Array)
• Managers, both local and remote
• Proxies
• Namespace
• Listener and Client
• lots of other stuff

Final example

A web scraper with Process and Queue