Python中如何在类内使用多进程执行类方法

有一个类似这样的类

class Foo:
def init(self):
self.x = ‘’
def func_1(self):
self.x += ‘f1’
def func_2(self):
self.x += ‘f2’
def process(self):
self.func_1()
self.func_2()

有办法让类内的 func_1 和 func_2 同时执行吗(func_1 和 func_2 实例为爬虫程序)?

eggper 1楼

读这个标题舌头快打结了。。

vueper 2楼

在Python类里用多进程跑类方法，核心问题是pickle序列化限制。类方法默认无法直接传给multiprocessing.Process，因为self.method会绑定实例对象。这里有几种实用方案：

方案1：使用multiprocessing.Pool.starmap配合静态方法

import multiprocessing as mp

class Processor:
    @staticmethod
    def worker(x):
        return x * x
    
    def run(self):
        with mp.Pool(processes=2) as pool:
            results = pool.map(self.worker, [1, 2, 3, 4])
        print(results)  # [1, 4, 9, 16]

if __name__ == '__main__':
    p = Processor()
    p.run()

方案2：包装函数绕过pickle限制

import multiprocessing as mp
from functools import partial

class Task:
    def process_item(self, item):
        return item.upper()
    
    def run(self):
        items = ['a', 'b', 'c']
        # 创建部分函数避免传递self
        worker = partial(self.process_item)
        
        with mp.Pool() as pool:
            results = pool.map(worker, items)
        print(results)  # ['A', 'B', 'C']

方案3：使用类方法+独立函数（最稳定）

import multiprocessing as mp

def external_worker(method, arg):
    """外部函数作为进程入口"""
    return method(arg)

class Calculator:
    @classmethod
    def compute(cls, x):
        return x ** 2
    
    def execute(self):
        with mp.Pool() as pool:
            # 传递类方法引用而非实例方法
            results = pool.starmap(external_worker, 
                                 [(self.compute, i) for i in range(5)])
        print(results)  # [0, 1, 4, 9, 16]

关键点：