# multiprocessing



John Melesky
(PDX Python, June 2010)
---
# In the 2.6 standard library

Otherwise, available from PyPI (>=2.4)
---
# A note

`multiprocessing` is a huge library, and deserving of a full-fledged talk (or several), rather than a module-of-the-month.

The manager functionality, for example, is elaborate and useful.

---
# A note

`multiprocessing` is a huge library, and deserving of a full-fledged talk (or several), rather than a module-of-the-month.

The manager functionality, for example, is elaborate and useful (as far as i can tell. i haven't used it. this is still just a motm talk).

---
# A different note

`multiprocessing` is complex, and occasionally slightly hackish. Expect the following:

- These examples may not work in the REPL
- Similar arguments may have different semantics depending on whether they're passed to `Process` or `Pool`
- It "works" on Windows
- Overall, YMMV

---
# Pre-multiprocessing

- fork/exec
- threading
---
# Pre-multiprocessing

- fork/exec
- threading (inspired by Java!)
---
# Process

[sample process code](code/proc1.py)
---
# Process

- name, pid, exitcode, authkey
- `start()`
- `run()`
- `join([timeout])`
- `is_alive()`
- `terminate()`
- daemon
---
# Pool
[sample pool code](code/proc2.py)

---
# Pool

- `apply(func[, args[, kwds]])`
- `apply_async(ditto, plus callback)`
- `map`, `map_async`, `imap`, `imap_unordered`
- `close()`
- `terminate()`
- `join()`

---
# Queue

[(nonworking) sample queue code](code/proc3.py)

---
# Queue

See Queue.Queue

- `empty()`, `full()`, `qsize()`
- `put(item[, block[, timeout]])`
- `put_nowait(item)`
- `get([block[, timeout]])`
- `get_nowait()`

---
# Pipe

[(semiworking) sample pipe code](code/proc4.py)

---
# Connection

- `send(obj)`
- `recv()`
- `fileno()`
- `close()`
- `poll([timeout])`
- `send_bytes`, `recv_bytes`, `recv_bytes_into`

---
# Stuff i'm not covering

- `Lock`
- shared memory maps (`Value`, `Array`)
- `Manager`s, both local and remote
- Proxies
- `Namespace`
- `Listener` and `Client`
- lots of other stuff

---
# Final example

A [web scraper](code/scraper.py) with Process and Queue

---