Python Scrapy 库
Spider/item/pipeline structure, and when Scrapy beats a requests loop.
Key PointsScrapy shines when you crawl many pages with dedupe, throttling and pipelines — for one page, requests is simpler.
Project Layout
Scrapy already handles scheduling, dedupe, throttling, retries and concurrency — you write what to crawl and how to parse.
Bash
pip install scrapy
scrapy startproject books
cd books && scrapy genspider list example.comText
books/
├── scrapy.cfg
└── books/
├── settings.py
├── items.py
├── pipelines.py
└── spiders/list.pyA Spider
parse yields either data (dict/Item) or new Requests (pagination). The framework feeds data to pipelines.
Python
import scrapy
class ListSpider(scrapy.Spider):
name = "list"
start_urls = ["https://example.com/books?page=1"]
def parse(self, response):
for card in response.css("article.book"):
yield {
"title": card.css("h3::text").get(default="").strip(),
"price": card.css(".price::text").get(default="").strip(),
}
next_page = response.css("a.next::attr(href)").get()
if next_page:
yield response.follow(next_page, self.parse)Bash
scrapy crawl list -O books.json # 直接输出 JSONOutput
{"title": "Python 编程", "price": "39.90"},
{"title": "Go 语言实战", "price": "45.00"},
{"title": "Rust 程序设计", "price": "49.00"}Politeness Settings
The default concurrency is 16 — unfriendly. AutoThrottle adapts to response time and beats a hard-coded delay.
Python
# settings.py
ROBOTSTXT_OBEY = True
CONCURRENT_REQUESTS_PER_DOMAIN = 4
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1.0
AUTOTHROTTLE_MAX_DELAY = 10.0ROBOTSTXT_OBEY = True is the default — do not disable it to save effort.项目结构
Scrapy 把「调度、去重、限速、重试、并发」都做好了，你只需要写「抓哪些 URL」和「怎么解析」。
Bash
pip install scrapy
scrapy startproject books
cd books && scrapy genspider list example.comText
books/
├── scrapy.cfg
└── books/
├── settings.py
├── items.py
├── pipelines.py
└── spiders/list.py一个爬虫
parse 收到响应后 yield 两种东西：数据（dict 或 Item）和新的请求（翻页）。框架负责把数据交给 pipeline。
Python
import scrapy
class ListSpider(scrapy.Spider):
name = "list"
start_urls = ["https://example.com/books?page=1"]
def parse(self, response):
for card in response.css("article.book"):
yield {
"title": card.css("h3::text").get(default="").strip(),
"price": card.css(".price::text").get(default="").strip(),
}
next_page = response.css("a.next::attr(href)").get()
if next_page:
yield response.follow(next_page, self.parse)Bash
scrapy crawl list -O books.json # 直接输出 JSONOutput
{"title": "Python 编程", "price": "39.90"},
{"title": "Go 语言实战", "price": "45.00"},
{"title": "Rust 程序设计", "price": "49.00"}礼貌抓取设置
默认并发是 16，对别人不友好。加上 AutoThrottle 让它根据响应时间自动调节，比写死 DOWNLOAD_DELAY 更稳。
Python
# settings.py
ROBOTSTXT_OBEY = True
CONCURRENT_REQUESTS_PER_DOMAIN = 4
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1.0
AUTOTHROTTLE_MAX_DELAY = 10.0ROBOTSTXT_OBEY = True 是默认值，别为了省事关掉。关掉它既不礼貌，也可能带来法律风险。