使用Python的pypiserver快速搭建内网离线pypi仓库实践
前言
本文介绍了如何快速搭建一个 pypiserver，通过自建 pypiserver，我们可以解决网络环境不好，或者离线无法安装 python 包的问题。如果结合最新的 GitLab CI/CD 和 pipenv 我相信各位还可以玩出更多的花样。
pypiserver - minimal PyPI server for use with pip/easy_install
更新记录
2018 年 04 月 12 日 - 初稿
阅读原文 - https://wsgzao.github.io/post/pypiserver/
扩展阅读
pypiserver - https://github.com/pypiserver/pypiserver
pypiserver 简介
pypiserver is a minimal PyPI compatible server for pip or easy_install. It is based on bottle and serves packages from regular directories. Wheels, bdists, eggs and accompanying PGP-signatures can be uploaded either with pip, setuptools, twine, pypi-uploader, or simply copied with scp.
pypiserver 服务端配置
如果你的 Linux 环境缺少 Python 2.7 可以参考我的文章直接离线升级至最新版本
Python 2.6 升级至 Python 2.7 的实践心得 https://wsgzao.github.io/post/python-2-6-update-to-2-7/
pypiserver > 1.2.x works with python 2.7 and 3.3+ or pypy. Older python-versions may still work, but they are not tested. For legacy python versions, use pypiserver-1.1.x series.
# 替换 pip 为阿里云，感概豆瓣的时代已经过去
tee ~/.pip/pip.conf <<-'EOF'
[global]
index-url = https://mirrors.aliyun.com/pypi/simple/
[install]
trusted-host= mirrors.aliyun.com
EOF
直接在线安装 pypiserver
pip install pypiserver
离线下载 pypiserver
mkdir /tmp/pypiserver
pip install -d /tmp/pypiserver/ pypiserver
Copy packages into this directory.
mkdir ~/packages
Copy some packages into your ~/packages folder and then get your pypiserver up and running:
pypi-server -p 8080 ~/packages &
pypiserver 客户端配置
## Download and Install hosted packages.
pip install --extra-index-url http://localhost:8080/simple/ ...
or
pip install --extra-index-url http://localhost:8080
Search hosted packages
pip search --index http://localhost:8080 …
个人推荐的配置
tee ~/.pip/pip.conf <<-‘EOF’
[global]
index-url = http://172.28.70.126/simple
extra-index-url = https://mirrors.aliyun.com/pypi/simple/
[install]
trusted-host = 172.28.70.126
EOF
pypiserver 进阶配置
pypiserver Running as a systemd service
https://github.com/pypiserver/pypiserver#running-as-a-systemd-service
Adjusting the paths and adding this file as pypiserver.service into your systemd/system directory will allow management of the pypiserver process with systemctl, e.g. systemctl start pypiserver.
More useful information about systemd can be found at https://www.digitalocean.com/community/tutorials/how-to-use-systemctl-to-manage-systemd-services-and-units
# 安装需要的包
yum install nginx -y
pip install passlib pypiserver gunicorn
创建 pypiserver 服务方便服务启停管理
tee /usr/lib/systemd/system/pypiserver.service <<-‘EOF’
[Unit]
Description=gunicorn daemon
After=network.target
[Service]
PIDFile=/run/pypiserver.pid
ExecStart=/usr/local/bin/gunicorn -w16
–pid /run/pypiserver.pid
-b :10012
‘pypiserver:app(root="/var/www/pypi")’
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID
[Install]
WantedBy=multi-user.target
EOF
Warning: pypiserver.service changed on disk. Run ‘systemctl daemon-reload’ to reload units.
systemctl daemon-reload
启动 pypiserver 服务
systemctl enable pypiserver.service
systemctl start pypiserver.service
systemctl status pypiserver.service
停止 pypiserver 服务
systemctl disable pypiserver.service
systemctl stop pypiserver.service
systemctl status pypiserver.service
[root@centos7 run]# systemctl status pypiserver.service
● pypiserver.service - gunicorn daemon
Loaded: loaded (/usr/lib/systemd/system/pypiserver.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2018-04-13 15:14:08 CST; 859ms ago
Main PID: 22524 (gunicorn)
CGroup: /system.slice/pypiserver.service
├─22524 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
├─22530 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
├─22531 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
├─22532 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
├─22533 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
├─22534 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
├─22535 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
├─22536 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
├─22537 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
├─22538 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
├─22539 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
└─22540 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22531] [INFO] Booting worker with pid: 22531
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22532] [INFO] Booting worker with pid: 22532
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22533] [INFO] Booting worker with pid: 22533
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22534] [INFO] Booting worker with pid: 22534
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22535] [INFO] Booting worker with pid: 22535
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22536] [INFO] Booting worker with pid: 22536
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22537] [INFO] Booting worker with pid: 22537
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22538] [INFO] Booting worker with pid: 22538
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22539] [INFO] Booting worker with pid: 22539
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22540] [INFO] Booting worker with pid: 22540
[root@centos7 run]# netstat -tnlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1/systemd
tcp 0 0 192.168.122.1:53 0.0.0.0:* LISTEN 1517/dnsmasq
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 977/sshd
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 978/cupsd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1383/master
tcp 0 0 127.0.0.1:6011 0.0.0.0:* LISTEN 19378/sshd: root@pt
tcp 0 0 0.0.0.0:10012 0.0.0.0:* LISTEN 22524/python
tcp6 0 0 :::111 :::* LISTEN 1/systemd
tcp6 0 0 :::22 :::* LISTEN 977/sshd
tcp6 0 0 ::1:631 :::* LISTEN 978/cupsd
tcp6 0 0 ::1:25 :::* LISTEN 1383/master
tcp6 0 0 ::1:6011 :::* LISTEN 19378/sshd: root@pt
检查 pypiserver 服务
cd /var/www/pypi
向仓库中添加 python package
[root@centos7 pypi]# pip download pypiserver
Collecting pypiserver
Downloading https://mirrors.aliyun.com/pypi/packages/d7/78/5772432dad2b9e754ab92f4d301fa507069b9decc8c943c1b18c2043ff4f/pypiserver-1.2.1-py2.py3-none-any.whl (83kB)
100% |████████████████████████████████| 92kB 643kB/s
Saved ./pypiserver-1.2.1-py2.py3-none-any.whl
Successfully downloaded pypiserver
[root@centos7 pypi]# ll
total 84
-rw-r–r-- 1 root root 83529 Apr 13 14:55 pypiserver-1.2.1-py2.py3-none-any.whl
搜索刚才下载的 package
[root@centos7 pypi]# pip search -i http://127.0.0.1:10012 pypiserver
pypiserver (1.2.1) - 1.2.1
INSTALLED: 1.2.1 (latest)
配置 nginx 做反向代理
tee /etc/nginx/conf.d/pypi.conf <<-‘EOF’
upstream pypiserver {
server 127.0.0.1:10012;
}
server {
listen 10087;
disable any limits to avoid HTTP 413 for large package uploads
client_max_body_size 0;
location / {
proxy_pass http://pypiserver/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# When setting up pypiserver behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering off;
proxy_request_buffering off;
}
location /packages/ {
alias /var/www/pypi; # static file
}
}
EOF
启动 nginx
systemctl enable nginx
systemctl start nginx
systemctl status nginx
检查 nginx 服务
pip search -i http://172.28.79.126:10087 pypiserver
pypiserver (1.2.1) - 1.2.1
INSTALLED: 1.2.1 (latest)
使用Python的pypiserver快速搭建内网离线pypi仓库实践
# 1. 安装pypiserver
# 使用pip直接安装（需要联网环境）
pip install pypiserver
# 2. 创建包存储目录
mkdir ~/packages
# 3. 启动基础服务（最简单的方式）
pypi-server -p 8080 ~/packages
# 4. 更完整的启动脚本（带认证和更多配置）
# 创建配置文件 pypiserver_config.py
import os
from pypiserver import app
# 配置参数
packages_dir = os.path.expanduser('~/packages')
password_file = os.path.expanduser('~/.htpasswd')
# 配置字典
config = {
'root': packages_dir,
'authenticate': ['download', 'upload'], # 下载和上传都需要认证
'passwords': password_file,
'port': 8080,
'host': '0.0.0.0', # 监听所有网络接口
'server': 'auto', # 自动选择服务器
'fallback_url': 'https://pypi.org/simple/' # 找不到包时回退到官方源
}
# 5. 使用gunicorn部署（生产环境推荐）
# 安装gunicorn: pip install gunicorn
# 启动命令：
# gunicorn -w 4 -b 0.0.0.0:8080 "pypiserver:app(root='~/packages')"
# 6. 创建认证文件（如果需要）
# 安装apache2-utils或使用htpasswd命令
# htpasswd -c ~/.htpasswd username
# 或者使用Python生成：
import base64
from passlib.apache import HtpasswdFile
ht = HtpasswdFile("~/.htpasswd")
ht.set_password("user1", "password123")
ht.save()
# 7. 上传包到私有仓库
# 安装twine: pip install twine
# 配置 ~/.pypirc
"""
[distutils]
index-servers =
local
[local]
repository: http://localhost:8080
username: your_username
password: your_password
"""
# 上传包：
# twine upload --repository local ~/packages/*
# 8. 客户端使用私有源
# 方式1：临时使用
pip install --index-url http://localhost:8080/simple/ some-package
# 方式2：配置pip.conf
"""
[global]
index-url = http://localhost:8080/simple/
trusted-host = localhost
"""
# 9. 完整的管理脚本示例
#!/usr/bin/env python3
"""
pypi_manager.py - 私有仓库管理脚本
"""
import subprocess
import sys
import os
class PyPIServerManager:
def __init__(self, port=8080, package_dir='~/packages'):
self.port = port
self.package_dir = os.path.expanduser(package_dir)
def start(self):
"""启动服务器"""
cmd = f"pypi-server -p {self.port} {self.package_dir}"
subprocess.Popen(cmd, shell=True)
print(f"服务器已启动: http://localhost:{self.port}")
def upload_package(self, package_path):
"""上传单个包"""
if not os.path.exists(package_path):
print(f"文件不存在: {package_path}")
return
import shutil
shutil.copy(package_path, self.package_dir)
print(f"已上传: {os.path.basename(package_path)}")
def list_packages(self):
"""列出所有包"""
if os.path.exists(self.package_dir):
packages = os.listdir(self.package_dir)
for pkg in packages:
print(f" - {pkg}")
else:
print("包目录为空")
if __name__ == "__main__":
manager = PyPIServerManager()
if len(sys.argv) > 1:
if sys.argv[1] == "start":
manager.start()
elif sys.argv[1] == "upload" and len(sys.argv) > 2:
manager.upload_package(sys.argv[2])
elif sys.argv[1] == "list":
manager.list_packages()
else:
print("用法: python pypi_manager.py [start|upload <file>|list]")
# 10. Docker部署方式（可选）
"""
# Dockerfile
FROM python:3.9-slim
RUN pip install pypiserver
RUN mkdir /data/packages
VOLUME /data/packages
EXPOSE 8080
CMD ["pypi-server", "-p", "8080", "/data/packages"]
# 构建和运行：
# docker build -t pypiserver .
# docker run -d -p 8080:8080 -v ~/packages:/data/packages pypiserver
"""
# 关键点总结：
# 1. 安装pypiserver：pip install pypiserver
# 2. 创建包目录：mkdir ~/packages
# 3. 启动服务：pypi-server -p 8080 ~/packages
# 4. 上传包：直接复制文件到包目录或使用twine
# 5. 客户端使用：pip install --index-url http://your-server:8080/simple/ package-name
# 一句话建议：用pypiserver搭建私有仓库就是装包、建目录、启动服务三步走。
这代码直接复制就能用，从安装到部署都齐了。
我记得当初测试 pypiserver 对于本地没有缓存的 pip 包的 pip install 请求会返回一个官网(默认)的源下载链接。对于无法访问公网的服务器来说这个 pip install 请求不会生效。。。
后来采用 docker 启的 devpi 服务完美解决的。
2.6 升级到 2.7 居然还能有心得？？
python 的安装做的有那么差吗？
python 还会完善 yum 的设置？越俎代庖还是你没搞清楚这俩东西没关联？
为什么那么爱好“离线包”、“上传到网盘”？
你制作的离线包，你会继续投入精力去持续更新吗？已经放出去的旧版本链接，你能召回吗？
#4 Blog 本身记录的是一种方法和解决问题的思路，很少有人可以持续更新技术类笔记，只是作为一个参考，我本来也觉得不是很复杂的东西但有时候就是会采坑，所以记录一点简单实用的，很复杂的技术自己放在笔记里和公司内部团队共享就好了
#3 pip search --index http://localhost:8080，我们内部用 GitLab+Jenkins 做团队开发的时候是通过 fabfile 写在脚本里执行的，等后续研究透彻 GitLab CI/CD + pipenv 的新方法后再做分享
问题是，你的“因为 Python 2.7.13 以后版本会自动完善 yum 配置，所以不必参考以前的网上文章去修改其他地方”既没有正确的说明（你以为的） python 和 yum 的关系，也没说“以前网上文章”哪些内容是不再需要的，你这样的写法并不能起到记录的作用
#7 Sorry，我点我确实没有做好，因为在 Python 官方安装的方法中没有看到有 FQ 提示，而 Google 搜索到的大部分信息都是通过修改软连接修复，将#!/usr/bin/python 修改为 #!/usr/bin/python2.6，我也没想太多，就以实际情况为主，没有去思考深沉次的原因
#9 从路径上来看应该是独立的，官方倒是也没有说明优化了哪些安装的方式，我实际测试从 python2.6 升级至 python2.7.13 没有问题，然后最近又升级到 python2.7.14 也是直接编译安装，相当顺利，后续估计要开始测试 python2.7 和 python3.6 使用 pipenv 共存的问题，估计又要折腾好久了bash<br>test101JQ/root#whereis python<br>python: /usr/bin/python2.6 /usr/bin/python /usr/lib/python2.6 /usr/lib64/python2.6 /usr/local/bin/python2.7 /usr/local/bin/python2.7-config /usr/local/bin/python /usr/local/lib/python2.7 /usr/include/python2.6 /usr/share/man/man1/python.1.gz<br>test101JQ/root#cat /usr/bin/yum<br>#!/usr/bin/python<br><br>test101JQ/root#find / -name python<br>/usr/share/doc/m2crypto-0.20.2/demo/ZopeX3/install_dir/lib/python<br>/usr/share/doc/m2crypto-0.20.2/demo/Zope27/install_dir/lib/python<br>/usr/share/doc/m2crypto-0.20.2/demo/Zope/lib/python<br>/usr/share/gdb/python<br>/usr/local/bin/python<br>/usr/bin/python<br>/root/.local/share/virtualenvs/pipenv-qb5OO9ES/bin/python<br>/root/Python-2.7.14-pip/Python-2.7.14/python<br>test101JQ/root#find / -name site-packages<br>/usr/lib/python2.6/site-packages<br>/usr/local/lib/python2.7/site-packages<br>/usr/lib64/python2.6/site-packages<br>/root/.local/share/virtualenvs/pipenv-qb5OO9ES/lib/python2.7/site-packages<br>/root/Python-2.7.14-pip/Python-2.7.14/Lib/site-packages<br><br>
