Python中使用requests替代scrapy下载器速度很慢，如何优化？

class MyDownloaderMiddleware(object):
def process_request(self, request, spider):
    res = requests.get(request.url)
    return HtmlResponse(request.url, body=res.content, encoding='utf-8', request=request)

scrapy 的下载器调教起来不如 requests 方便，所以想用 requests 替代

itying888 1楼

因为你这是一个同步操作,用 deferreds

yibo5220 2楼

用requests替代Scrapy下载器慢是正常的，Scrapy内置了异步并发和连接复用机制。要提速，直接用aiohttp搞异步并发最直接。

import aiohttp
import asyncio
from time import time

async def fetch(session, url):
    async with session.get(url) as response:
        return await response.text()

async def main():
    urls = ['http://example.com/page{}'.format(i) for i in range(100)]
    
    connector = aiohttp.TCPConnector(limit=50)  # 调大并发连接数
    timeout = aiohttp.ClientTimeout(total=30)
    
    async with aiohttp.ClientSession(connector=connector, timeout=timeout) as session:
        tasks = [fetch(session, url) for url in urls]
        results = await asyncio.gather(*tasks, return_exceptions=True)
        
        for url, result in zip(urls, results):
            if isinstance(result, Exception):
                print(f"Failed {url}: {result}")
            else:
                print(f"Got {url}: {len(result)} chars")

if __name__ == '__main__':
    start = time()
    asyncio.run(main())
    print(f"Time: {time() - start:.2f}s")

关键点：

用aiohttp替代requests：原生支持异步，比requests快一个数量级
调大TCPConnector的limit参数：默认是100，根据机器性能可以调到200-500
设置合理的timeout：避免单个请求卡住整个流程
批量使用asyncio.gather：一次性并发多个请求

如果非要坚持用requests，可以用ThreadPoolExecutor搞线程池，但效果不如aiohttp：

from concurrent.futures import ThreadPoolExecutor
import requests

def fetch(url):
    return requests.get(url).text

with ThreadPoolExecutor(max_workers=50) as executor:
    results = list(executor.map(fetch, urls))

简单说就是换异步库或者上线程池。