Python爬虫如何应对拉勾网的反爬机制？

拉勾网的反爬确实比较严，主要是动态加载和请求头校验。我常用的方案是用Selenium配合一些等待策略，这里给个完整示例：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

# 设置Chrome选项
options = webdriver.ChromeOptions()
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)

driver = webdriver.Chrome(options=options)
driver.execute_cdp_cmd('Page.addScriptToEvaluateOnNewDocument', {
    'source': '''
    Object.defineProperty(navigator, 'webdriver', {
        get: () => undefined
    })
    '''
})

try:
    driver.get('https://www.lagou.com/jobs/list_python')
    
    # 等待页面加载
    wait = WebDriverWait(driver, 10)
    job_list = wait.until(
        EC.presence_of_element_located((By.CLASS_NAME, "job-list"))
    )
    
    # 滚动加载更多
    for _ in range(3):
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        time.sleep(2)
    
    # 提取数据
    jobs = driver.find_elements(By.CLASS_NAME, "con_list_item")
    for job in jobs:
        title = job.find_element(By.TAG_NAME, "h3").text
        company = job.find_element(By.CLASS_NAME, "company_name").text
        salary = job.find_element(By.CLASS_NAME, "money").text
        print(f"{title} | {company} | {salary}")
        
finally:
    driver.quit()

关键点：1）用CDP命令隐藏webdriver特征；2）合理设置等待时间避免被识别；3）模拟人类滚动行为。如果数据量不大，这个方案够用了。

真要大规模爬的话得研究他们的接口，不过那又是另一回事了。