如何用Python提高文件上传速度

需要提供一个接口，接口接收客户端上传的文件，然后进行特定处理，把结果返回客户端。

现在的问题是,如何提高上传的速度？提高服务器带宽?使用压缩?

服务端可以做哪些工作提高上传的速度？

如何用Python提高文件上传速度

ionicwang 1楼

服务端的下载带宽就是客户端最大可用的上传带宽。不过在国内来说，客户端上传都好不到哪里去，除非是公司上下对待的商业宽带。

songsunli 2楼

对于用Python提高文件上传速度，核心是减少网络往返和利用并行传输。这里给你几个实用的方案：

1. 使用流式上传和分块传输

import requests
from requests_toolbelt.multipart.encoder import MultipartEncoder

# 流式上传大文件
def upload_large_file(file_path, url, chunk_size=8192):
    with open(file_path, 'rb') as f:
        encoder = MultipartEncoder(
            fields={'file': ('filename', f, 'application/octet-stream')}
        )
        
        headers = {'Content-Type': encoder.content_type}
        
        # 分块上传
        response = requests.post(
            url, 
            data=encoder, 
            headers=headers,
            stream=True
        )
    return response

2. 多线程并行上传

from concurrent.futures import ThreadPoolExecutor
import os

def upload_chunk(chunk_data, chunk_num, url):
    files = {'chunk': (f'chunk_{chunk_num}', chunk_data)}
    return requests.post(url, files=files)

def parallel_upload(file_path, url, max_workers=4, chunk_size=1024*1024):
    chunks = []
    with open(file_path, 'rb') as f:
        while True:
            chunk = f.read(chunk_size)
            if not chunk:
                break
            chunks.append(chunk)
    
    with ThreadPoolExecutor(max_workers=max_workers) as executor:
        futures = []
        for i, chunk in enumerate(chunks):
            future = executor.submit(upload_chunk, chunk, i, url)
            futures.append(future)
        
        results = [f.result() for f in futures]
    return results

3. 使用aiohttp进行异步上传

import aiohttp
import asyncio

async def async_upload(file_path, url, chunk_size=8192):
    async with aiohttp.ClientSession() as session:
        with open(file_path, 'rb') as f:
            data = aiohttp.FormData()
            data.add_field('file', f, filename=os.path.basename(file_path))
            
            async with session.post(url, data=data) as response:
                return await response.text()

# 使用示例
# asyncio.run(async_upload('large_file.zip', 'https://example.com/upload'))

4. 压缩后再上传（适合文本类文件）

import gzip
import io

def upload_compressed(file_path, url):
    with open(file_path, 'rb') as f:
        original_data = f.read()
    
    # 压缩数据
    compressed = gzip.compress(original_data)
    
    # 使用BytesIO包装压缩数据
    compressed_file = io.BytesIO(compressed)
    
    files = {'file': ('compressed.gz', compressed_file)}
    response = requests.post(url, files=files)
    return response

选择建议：