关于Python中Scrapy框架signals的用法请教

Scrapy的signals机制是框架的核心扩展点，让你能在爬虫生命周期中插入自定义逻辑。它基于pydispatch实现，通过信号/槽机制工作。

最常用的几个信号：

• engine_started/engine_stopped：引擎启动/停止时触发
• spider_opened/spider_closed：爬虫打开/关闭时触发
• item_scraped：item被抓取后触发
• request_scheduled/response_received：请求调度和响应接收时触发

基本用法示例：

from scrapy import signals
from scrapy.crawler import Crawler

class MySpider(scrapy.Spider):
    name = 'example'
    
    @classmethod
    def from_crawler(cls, crawler, *args, **kwargs):
        spider = super().from_crawler(crawler, *args, **kwargs)
        # 连接信号处理器
        crawler.signals.connect(spider.spider_opened, signal=signals.spider_opened)
        crawler.signals.connect(spider.spider_closed, signal=signals.spider_closed)
        crawler.signals.connect(spider.item_scraped, signal=signals.item_scraped)
        return spider
    
    def spider_opened(self, spider):
        print(f"爬虫 {spider.name} 启动了")
    
    def spider_closed(self, spider):
        print(f"爬虫 {spider.name} 关闭了")
    
    def item_scraped(self, item, response, spider):
        print(f"抓到item: {item}")
    
    def start_requests(self):
        yield scrapy.Request('http://example.com', callback=self.parse)
    
    def parse(self, response):
        yield {'title': response.css('title::text').get()}

也可以在扩展(Extension)中使用：

from scrapy import signals
from scrapy.exceptions import NotConfigured

class StatsExtension:
    def __init__(self, stats):
        self.stats = stats
        self.item_count = 0
    
    @classmethod
    def from_crawler(cls, crawler):
        ext = cls(crawler.stats)
        crawler.signals.connect(ext.spider_opened, signal=signals.spider_opened)
        crawler.signals.connect(ext.item_scraped, signal=signals.item_scraped)
        return ext
    
    def spider_opened(self, spider):
        self.stats.set_value('start_time', time.time())
    
    def item_scraped(self, item, response, spider):
        self.item_count += 1
        self.stats.set_value('items_scraped', self.item_count)

关键点：信号处理器接收的参数由具体信号决定，需要查看Scrapy文档确认。比如item_scraped接收(item, response, spider)三个参数，而spider_opened只接收(spider)一个参数。

信号机制让Scrapy的扩展性很强，但别滥用，过度使用会影响性能。

总结：用from_crawler连接信号，根据需求选择合适的信号点。