Python selenium 库
Drivers, waits, selectors, and the difference between implicit and explicit waits.
Key PointsNever use `time.sleep` to wait for elements — use explicit `WebDriverWait` with a condition.
Setup
Selenium 4 manages drivers automatically — no manual chromedriver download. Headless mode is what you want on a server.
Bash
pip install seleniumPython
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
opts = Options()
opts.add_argument("--headless=new")
opts.add_argument("--no-sandbox")
opts.add_argument("--window-size=1440,900")
driver = webdriver.Chrome(options=opts)
driver.get("https://example.com")
print(driver.title)
driver.quit()Output
Example DomainExplicit Waits
Pages render asynchronously. Implicit waits only cover element lookup; conditions like visible/clickable need explicit waits.
Python
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, timeout=10)
btn = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.submit")))
btn.click()
rows = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "table tbody tr")))
print(len(rows))Output
12Selectors and Input
| Strategy | Example | Note |
|---|---|---|
| By.ID | "login" | Fastest, use when available |
| By.CSS_SELECTOR | "form input[name=user]" | Most flexible |
| By.XPATH | "//button[text()='提交']" | Text matching |
| By.CLASS_NAME | "item" | Single class only |
Python
driver.find_element(By.ID, "user").send_keys("alice")
driver.find_element(By.ID, "pwd").send_keys("secret")
driver.find_element(By.CSS_SELECTOR, "button[type=submit]").click()Read robots.txt and the terms first. Aggressive automation puts real load on the target and may violate its ToS.
环境准备
Selenium 4 自带驱动管理，不再需要手动下载 chromedriver。无头模式适合服务器上跑。
Bash
pip install seleniumPython
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
opts = Options()
opts.add_argument("--headless=new")
opts.add_argument("--no-sandbox")
opts.add_argument("--window-size=1440,900")
driver = webdriver.Chrome(options=opts)
driver.get("https://example.com")
print(driver.title)
driver.quit()Output
Example Domain显式等待
页面是异步渲染的，元素出现需要时间。隐式等待只对「查找元素」生效，遇到「元素可见/可点击」这类条件就不够用。
Python
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, timeout=10)
btn = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.submit")))
btn.click()
rows = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "table tbody tr")))
print(len(rows))Output
12选择器与输入
| 方式 | 示例 | 说明 |
|---|---|---|
| By.ID | "login" | 最快，优先用 |
| By.CSS_SELECTOR | "form input[name=user]" | 最灵活 |
| By.XPATH | "//button[text()='提交']" | 按文本匹配 |
| By.CLASS_NAME | "item" | 只支持单个 class |
Python
driver.find_element(By.ID, "user").send_keys("alice")
driver.find_element(By.ID, "pwd").send_keys("secret")
driver.find_element(By.CSS_SELECTOR, "button[type=submit]").click()爬虫前先看 robots.txt 与站点条款。高频自动化访问会给对方造成真实压力，也可能违反服务协议。