Python 爬虫

Fetch and Parse

Fetch HTML with requests, extract with BeautifulSoup CSS selectors. Prefer the lxml parser — much faster than the built-in one.

Bash

pip install requests beautifulsoup4 lxml

Python

import requests
from bs4 import BeautifulSoup

headers = {"User-Agent": "Mozilla/5.0 (compatible; MyBot/1.0)"}
resp = requests.get("https://example.com/books", headers=headers, timeout=10)
resp.raise_for_status()

soup = BeautifulSoup(resp.text, "lxml")
for item in soup.select("article.book")[:3]:
    title = item.select_one("h3").get_text(strip=True)
    price = item.select_one(".price").get_text(strip=True)
    print(title, price)

Output

Python 编程 39.90
Go 语言实战 45.00
Rust 程序设计 49.00

Pagination

Pagination is either a URL param or a next button. The URL param is more robust — just loop until empty.

Python

import time

rows = []
for page in range(1, 4):
    r = requests.get(f"https://example.com/books?page={page}", headers=headers, timeout=10)
    soup = BeautifulSoup(r.text, "lxml")
    items = soup.select("article.book")
    if not items:
        break
    rows += [i.select_one("h3").get_text(strip=True) for i in items]
    time.sleep(1)          # 限速：每秒最多 1 个请求

print(len(rows))

Output

Anti-Scraping Reality

Obstacle	Typical handling
Dynamic JS rendering	Use Playwright/Selenium, or find the JSON API
Rate limiting	Sleep + rotate IP + respect Retry-After
Login wall	Use a session with cookies, never store others' credentials
Captcha	Stop — usually means you are not welcome

Look for the JSON API first: many "dynamic pages" are just an XHR returning JSON — far more stable than DOM parsing.

抓取与解析

先用 requests 拿到 HTML，再交给 BeautifulSoup 用 CSS 选择器提取。解析库推荐 lxml，比内置解析器快很多。

Bash

pip install requests beautifulsoup4 lxml

Python

import requests
from bs4 import BeautifulSoup

headers = {"User-Agent": "Mozilla/5.0 (compatible; MyBot/1.0)"}
resp = requests.get("https://example.com/books", headers=headers, timeout=10)
resp.raise_for_status()

soup = BeautifulSoup(resp.text, "lxml")
for item in soup.select("article.book")[:3]:
    title = item.select_one("h3").get_text(strip=True)
    price = item.select_one(".price").get_text(strip=True)
    print(title, price)

Output

Python 编程 39.90
Go 语言实战 45.00
Rust 程序设计 49.00

翻页

翻页一般有两种：URL 参数（?page=2）和「下一页」按钮。前者更稳，直接循环；后者要判断按钮是否还存在。

Python

import time

rows = []
for page in range(1, 4):
    r = requests.get(f"https://example.com/books?page={page}", headers=headers, timeout=10)
    soup = BeautifulSoup(r.text, "lxml")
    items = soup.select("article.book")
    if not items:
        break
    rows += [i.select_one("h3").get_text(strip=True) for i in items]
    time.sleep(1)          # 限速：每秒最多 1 个请求

print(len(rows))

Output

反爬的现实

障碍	常见处理
JS 动态渲染	用 Playwright/Selenium，或直接找 JSON 接口
限流	sleep + 换 IP + 遵守 Retry-After
登录墙	用带 cookie 的 session，不要存他人凭据
验证码	停下来 —— 通常意味着对方不欢迎

优先找数据接口：打开 DevTools 的 Network 面板，很多「动态页面」其实是一个返回 JSON 的 XHR，直接调它比解析 DOM 稳得多。

Python 爬虫

Fetch and Parse

Pagination

Anti-Scraping Reality

抓取与解析

翻页

反爬的现实

Friends & Partners