Python中Scrapy爬虫的简单工程设计方法

前排支持！

对于Scrapy爬虫的工程设计，核心是遵循模块化原则。我通常这样组织项目结构：

# 项目结构示例
myproject/
├── scrapy.cfg
└── myproject/
    ├── __init__.py
    ├── items.py           # 数据模型定义
    ├── middlewares.py     # 中间件
    ├── pipelines.py       # 数据处理管道
    ├── settings.py        # 配置文件
    └── spiders/
        ├── __init__.py
        ├── base_spider.py # 基础爬虫类
        └── example_spider.py

基础爬虫类设计：

# spiders/base_spider.py
import scrapy
from scrapy.http import Request
from myproject.items import MyItem

class BaseSpider(scrapy.Spider):
    name = 'base'
    custom_settings = {
        'CONCURRENT_REQUESTS': 16,
        'DOWNLOAD_DELAY': 1,
    }
    
    def start_requests(self):
        """重写此方法定义起始请求"""
        yield Request(self.start_url, callback=self.parse_list)
    
    def parse_list(self, response):
        """解析列表页"""
        # 提取详情页链接
        detail_links = response.css('.detail-link::attr(href)').getall()
        for link in detail_links:
            yield Request(link, callback=self.parse_detail)
        
        # 翻页逻辑
        next_page = response.css('.next-page::attr(href)').get()
        if next_page:
            yield Request(next_page, callback=self.parse_list)
    
    def parse_detail(self, response):
        """解析详情页"""
        item = MyItem()
        item['title'] = response.css('h1::text').get()
        item['content'] = response.css('.content::text').getall()
        yield item

数据模型定义：

# items.py
import scrapy

class MyItem(scrapy.Item):
    title = scrapy.Field()
    content = scrapy.Field()
    url = scrapy.Field()
    crawl_time = scrapy.Field()

管道处理：

# pipelines.py
import json
from itemadapter import ItemAdapter

class JsonWriterPipeline:
    def open_spider(self, spider):
        self.file = open('output.json', 'w', encoding='utf-8')
    
    def close_spider(self, spider):
        self.file.close()
    
    def process_item(self, item, spider):
        line = json.dumps(ItemAdapter(item).asdict(), ensure_ascii=False)
        self.file.write(line + "\n")
        return item

配置设置：

# settings.py
BOT_NAME = 'myproject'
SPIDER_MODULES = ['myproject.spiders']
NEWSPIDER_MODULE = 'myproject.spiders'

ITEM_PIPELINES = {
    'myproject.pipelines.JsonWriterPipeline': 300,
}

# 遵守robots协议
ROBOTSTXT_OBEY = True

# 请求头设置
DEFAULT_REQUEST_HEADERS = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
}

具体爬虫实现：

# spiders/example_spider.py
from .base_spider import BaseSpider

class ExampleSpider(BaseSpider):
    name = 'example'
    start_url = 'https://example.com/list'
    
    def parse_detail(self, response):
        item = super().parse_detail(response)
        # 添加额外处理逻辑
        item['author'] = response.css('.author::text').get()
        return item

关键设计要点：1）使用基础爬虫类封装通用逻辑；2）清晰分离数据模型、爬虫逻辑和数据处理；3）合理配置爬取策略；4）保持代码的可扩展性。

总结：模块化设计让爬虫更易维护和扩展。