Python3 urllib
urllib.request 发 GET/POST 请求、设置请求头、解析 URL 与异常处理。
urllib.request
urllib.request.urlopen(url) opens a URL and returns a response object readable like a file. read() gives bytes that must be decoded using the charset from the headers.
import urllib.request
import urllib.parse
req = urllib.request.Request('https://example.com')
print(req.get_method())
print(req.full_url)
print(urllib.parse.quote('中文 空格'))
print(urllib.parse.urlencode({'q': 'python', 'n': 2}))GET
https://example.com
%E4%B8%AD%E6%96%87%20%E7%A9%BA%E6%A0%BC
q=python&n=2GET 请求
GET puts parameters in the query string and suits reads. Building it with urlencode percent-encodes automatically, so spaces and non-ASCII do not break.
import urllib.parse
base = 'https://httpbin.org/get'
query = urllib.parse.urlencode({'q': 'python', 'page': 2})
url = base + '?' + query
print(url)
parts = urllib.parse.urlparse(url)
print(parts.netloc)
print(urllib.parse.parse_qs(parts.query))https://httpbin.org/get?q=python&page=2
httpbin.org
{'q': ['python'], 'page': ['2']}POST 请求
POST sends data in the body, which must be encoded to bytes before passing as data=. Forms use application/x-www-form-urlencoded; JSON uses application/json.
import urllib.parse
import json
form = urllib.parse.urlencode({'user': 'tom', 'pwd': 'x'}).encode()
print(form)
payload = json.dumps({'user': 'tom'}).encode()
print(payload)
print(len(form) > 0, len(payload) > 0)b'user=tom&pwd=x'
b'{"user": "tom"}'
True True添加请求头
Many sites reject requests without a User-Agent. Build a Request(url, headers={...}, method=...) and hand it to urlopen.
import urllib.request
req = urllib.request.Request(
'https://example.com',
headers={'User-Agent': 'BackendWiki/1.0', 'Accept': 'text/html'},
method='GET',
)
print(req.get_header('User-agent'))
print(req.get_header('Accept'))
print(req.get_method())BackendWiki/1.0
text/html
GET异常处理
Two exceptions matter: URLError (DNS failure, connection refused) and HTTPError (4xx/5xx responses). The latter subclasses the former, so catch it first.
import urllib.error
print(issubclass(urllib.error.HTTPError, urllib.error.URLError))
try:
raise urllib.error.HTTPError('https://x', 404, 'Not Found', {}, None)
except urllib.error.HTTPError as e:
print("code:", e.code, "reason:", e.reason)
except urllib.error.URLError as e:
print("url error:", e.reason)True
code: 404 reason: Not Foundurllib.request
urllib.request.urlopen(url) 打开 URL 并返回响应对象，像文件一样可读。read() 得到字节，需要按响应头声明的编码解码。
import urllib.request
import urllib.parse
req = urllib.request.Request('https://example.com')
print(req.get_method())
print(req.full_url)
print(urllib.parse.quote('中文 空格'))
print(urllib.parse.urlencode({'q': 'python', 'n': 2}))GET
https://example.com
%E4%B8%AD%E6%96%87%20%E7%A9%BA%E6%A0%BC
q=python&n=2GET 请求
GET 把参数放在 URL 查询串里，适合读取数据。用 urlencode 拼查询串能自动做百分号编码，避免中文和空格出错。
import urllib.parse
base = 'https://httpbin.org/get'
query = urllib.parse.urlencode({'q': 'python', 'page': 2})
url = base + '?' + query
print(url)
parts = urllib.parse.urlparse(url)
print(parts.netloc)
print(urllib.parse.parse_qs(parts.query))https://httpbin.org/get?q=python&page=2
httpbin.org
{'q': ['python'], 'page': ['2']}POST 请求
POST 把数据放在请求体里，必须 encode() 成字节再传给 urlopen(data=...)。表单用 application/x-www-form-urlencoded，JSON 用 application/json。
import urllib.parse
import json
form = urllib.parse.urlencode({'user': 'tom', 'pwd': 'x'}).encode()
print(form)
payload = json.dumps({'user': 'tom'}).encode()
print(payload)
print(len(form) > 0, len(payload) > 0)b'user=tom&pwd=x'
b'{"user": "tom"}'
True True添加请求头
很多站点会拒绝没有 User-Agent 的请求。用 Request(url, headers={...}, method=...) 构造请求，再交给 urlopen。
import urllib.request
req = urllib.request.Request(
'https://example.com',
headers={'User-Agent': 'BackendWiki/1.0', 'Accept': 'text/html'},
method='GET',
)
print(req.get_header('User-agent'))
print(req.get_header('Accept'))
print(req.get_method())BackendWiki/1.0
text/html
GET异常处理
网络请求要防两类异常：URLError（DNS 失败、连接被拒）和 HTTPError（服务器返回 4xx/5xx）。后者是前者的子类，先捕获它。
import urllib.error
print(issubclass(urllib.error.HTTPError, urllib.error.URLError))
try:
raise urllib.error.HTTPError('https://x', 404, 'Not Found', {}, None)
except urllib.error.HTTPError as e:
print("code:", e.code, "reason:", e.reason)
except urllib.error.URLError as e:
print("url error:", e.reason)True
code: 404 reason: Not Found