Python中如何在Scrapy框架下实现MongoDB去重与下载管道管理

在Scrapy里用MongoDB做去重和管道管理，核心是自定义Item Pipeline和结合scrapy.dupefilters。下面给你一个完整的实现方案。

首先，确保安装了必要的包：

pip install pymongo scrapy

1. 去重过滤器 (DupeFilter) 在middlewares.py或单独文件（如dupefilter.py）中创建基于MongoDB的去重中间件：

# dupefilter.py
from scrapy.dupefilters import BaseDupeFilter
from pymongo import MongoClient
from scrapy.utils.request import request_fingerprint

class MongoDupeFilter(BaseDupeFilter):
    def __init__(self, mongo_uri, mongo_db, collection_name):
        self.client = MongoClient(mongo_uri)
        self.db = self.client[mongo_db]
        self.collection = self.db[collection_name]
        self.collection.create_index('fingerprint', unique=True)

    @classmethod
    def from_crawler(cls, crawler):
        settings = crawler.settings
        return cls(
            mongo_uri=settings.get('MONGO_URI', 'mongodb://localhost:27017'),
            mongo_db=settings.get('MONGO_DATABASE', 'scrapy_db'),
            collection_name=settings.get('DUPEFILTER_COLLECTION', 'seen_urls')
        )

    def request_seen(self, request):
        fp = request_fingerprint(request)
        try:
            self.collection.insert_one({'fingerprint': fp})
            return False
        except:
            return True

    def close(self, reason):
        self.client.close()

2. 下载管道 (Pipeline) 在pipelines.py中创建MongoDB存储管道：

# pipelines.py
from pymongo import MongoClient
from itemadapter import ItemAdapter

class MongoPipeline:
    def __init__(self, mongo_uri, mongo_db):
        self.mongo_uri = mongo_uri
        self.mongo_db = mongo_db

    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            mongo_uri=crawler.settings.get('MONGO_URI'),
            mongo_db=crawler.settings.get('MONGO_DATABASE', 'scrapy_db')
        )

    def open_spider(self, spider):
        self.client = MongoClient(self.mongo_uri)
        self.db = self.client[self.mongo_db]

    def close_spider(self, spider):
        self.client.close()

    def process_item(self, item, spider):
        collection_name = spider.name  # 或用item['collection']指定
        self.db[collection_name].insert_one(ItemAdapter(item).asdict())
        return item

3. 配置settings.py

# settings.py
MONGO_URI = 'mongodb://localhost:27017'
MONGO_DATABASE = 'scrapy_db'
DUPEFILTER_COLLECTION = 'seen_urls'

# 启用自定义去重过滤器
DUPEFILTER_CLASS = 'your_project.dupefilter.MongoDupeFilter'

# 启用管道并设置优先级
ITEM_PIPELINES = {
    'your_project.pipelines.MongoPipeline': 300,
}

工作原理：

• MongoDupeFilter会在爬取前检查请求指纹是否已存在于MongoDB，实现去重。
• MongoPipeline负责将清洗后的数据存储到MongoDB。
• 通过settings.py统一管理配置，便于维护。

一句话建议： 记得在MongoDB中为去重集合建立唯一索引提升性能。

v2ex 不支持 markdown…

https://gist.github.com/watsy0007/779c27fb0ceab283cc434b5eec10b7c4

封装了针对数据处理的公共方法.