Python3 多线程
What the GIL really blocks, when threads still help, and how to share data safely.
Key PointsThe GIL serialises bytecode execution, so threads speed up I/O-bound work, not CPU-bound work — use processes for the latter.
What the GIL Is
CPython has a global interpreter lock: only one thread executes Python bytecode at a time. Threads still help because most real work waits — on sockets, disks, databases — and the GIL is released during those waits.
| Workload | Threads | Processes | Why |
|---|---|---|---|
| HTTP / DB calls | Yes | Overkill | Waiting releases the GIL |
| File / disk I/O | Yes | Overkill | Same reason |
| Pure-Python CPU loops | No | Yes | GIL serialises bytecode |
| NumPy / C extensions | Partly | Yes | C code may release the GIL |
Thread Basics
Python
import threading
import time
def fetch(url):
time.sleep(0.5) # 模拟网络等待
print("done", url)
threads = [threading.Thread(target=fetch, args=(f"u{i}",)) for i in range(3)]
for t in threads:
t.start()
for t in threads:
t.join() # 等全部结束ThreadPoolExecutor
Prefer the executor API: it manages worker lifecycle, returns futures, and propagates exceptions. Do not spawn hundreds of raw threads.
Python
from concurrent.futures import ThreadPoolExecutor, as_completed
urls = [f"https://api.example.com/{i}" for i in range(10)]
with ThreadPoolExecutor(max_workers=8) as pool:
futures = {pool.submit(fetch, u): u for u in urls}
for fut in as_completed(futures):
url = futures[fut]
try:
fut.result()
except Exception as exc:
print("failed", url, exc)Races and Locks
Python
import threading
counter = 0
lock = threading.Lock()
def bump():
global counter
for _ in range(100_000):
with lock: # 没有锁，结果会小于 200000
counter += 1
ts = [threading.Thread(target=bump) for _ in range(2)]
[t.start() for t in ts]
[t.join() for t in ts]
print(counter)Producer / Consumer with queue
Python
import queue
import threading
q = queue.Queue(maxsize=100)
def producer():
for i in range(5):
q.put(i)
q.put(None) # 结束哨兵
def consumer():
while (item := q.get()) is not None:
print("got", item)
q.task_done()
threading.Thread(target=producer).start()
threading.Thread(target=consumer).start()queue.Queue is already thread-safe — do not wrap it in another lock. For CPU-bound work use ProcessPoolExecutor, or you will be slower than single-threaded.GIL 是什么
CPython 有一把全局解释器锁：同一时刻只有一个线程在执行 Python 字节码。线程仍然有用，因为真实工作大多在等待 —— 等 socket、等磁盘、等数据库 —— 等待期间 GIL 是释放的。
| 负载类型 | 多线程 | 多进程 | 原因 |
|---|---|---|---|
| HTTP、数据库调用 | 有效 | 没必要 | 等待时会释放 GIL |
| 文件与磁盘 I/O | 有效 | 没必要 | 同上 |
| 纯 Python 计算循环 | 无效 | 有效 | GIL 串行化字节码 |
| NumPy、C 扩展 | 部分有效 | 有效 | C 代码可能释放 GIL |
线程基础
Python
import threading
import time
def fetch(url):
time.sleep(0.5) # 模拟网络等待
print("done", url)
threads = [threading.Thread(target=fetch, args=(f"u{i}",)) for i in range(3)]
for t in threads:
t.start()
for t in threads:
t.join() # 等全部结束线程池
优先用线程池 API：它管理线程生命周期、返回 future、能传播异常。不要直接开几百个裸线程。
Python
from concurrent.futures import ThreadPoolExecutor, as_completed
urls = [f"https://api.example.com/{i}" for i in range(10)]
with ThreadPoolExecutor(max_workers=8) as pool:
futures = {pool.submit(fetch, u): u for u in urls}
for fut in as_completed(futures):
url = futures[fut]
try:
fut.result()
except Exception as exc:
print("failed", url, exc)竞态与锁
Python
import threading
counter = 0
lock = threading.Lock()
def bump():
global counter
for _ in range(100_000):
with lock: # 没有锁，结果会小于 200000
counter += 1
ts = [threading.Thread(target=bump) for _ in range(2)]
[t.start() for t in ts]
[t.join() for t in ts]
print(counter)用 queue 做生产者消费者
Python
import queue
import threading
q = queue.Queue(maxsize=100)
def producer():
for i in range(5):
q.put(i)
q.put(None) # 结束哨兵
def consumer():
while (item := q.get()) is not None:
print("got", item)
q.task_done()
threading.Thread(target=producer).start()
threading.Thread(target=consumer).start()queue.Queue 本身就是线程安全的 —— 不要再套一层锁。CPU 密集型请用 ProcessPoolExecutor，否则比单线程还慢。