332 views
 owned this note
# Bootstrapping Python 2022-10-31 Attendees: * felixonmars * FFY00 * dvzrv * jelle ## Agenda Some of the topics have been highlighted in https://wiki.archlinux.org/title/User:Davezerave/Python_Bootstrapping ### Building Python 3.11 Already build 3.11 and bootstrapped packages https://pkgbuild.com/~jelle/python3.11/ ### Bootstrap Python 3.11 For bootstrapping the basic packages we currently need: * python-build * [python-flit-core](https://github.com/pypa/flit/tree/main/flit_core) ([self-bootstrapping](https://github.com/pypa/flit/blob/7b13fb2d1425411cc0c03bb9ed9d49264c1e4dbb/flit_core/pyproject.toml#L1-L4)) (python-tomli or backport https://github.com/pypa/flit/commit/dba0f317c52d3c9fa29792a0beb0c0abf4476a23) * python-setuptools (ensurepip hack: https://github.com/archlinux/svntogit-packages/commit/8ac0104aa4758689c2aaf938a1e2043466cf3d03) * python-installer * [python-poetry-core](https://github.com/python-poetry/poetry-core) ([self-bootstrapping](https://github.com/python-poetry/poetry-core/blob/dd1245859e8a2349d26d66a0ebdcb82ff3fc09db/pyproject.toml#L118-L121)) * [python-pdm-pep517](https://github.com/pdm-project/pdm-pep517) ([self-bootstrapping](https://github.com/pdm-project/pdm-pep517/blob/cb6eefedb163eae76dd4ba9392608585fe9193b1/pyproject.toml#L52-L55) if vendored!) * [python-hatch](https://github.com/pypa/hatch/) ([self-bootstrapping](https://github.com/pypa/hatch/blob/master/backend/pyproject.toml)) (there may be hidden dependencies, reported only via the runtime `get_requires_for_build_wheel` hook) bootstrap currently provides: * build * flit_core * installer * pep517 * setuptools * tomli * wheel ### Bootstrap build backends Some build backends may require bootstraping themselves, if one of their dependencies use said backend to build. #### List of build backends in our repos: - python-setuptools - python-flit-core + python-flit - python-poetry-core + python-poetry - python-pdm-pep517 + python-pdm - python-hatchling + python-hatch - python-sphinx-theme-builder - meson-python ### Bootstrap workflow suggestion * Use python-bootstrap to create each(?!) bootstrap package (e.g. python-build-bootstrap providing python-build, etc.) in [staging] * `python -m bootstrap.install $wheel --destdir "$pkgdir"` * Build all build and runtime requirements of the initial bootstrap packages * Build all initial bootstrap packages properly and replace the initial bootstrap packages * Commence with rebuild as normal https://github.com/jelly/python-bootstrap/tree/PKGBUILD ``` # Maintainer pkgbase=python-bootstrap pkgname=(python-build python-installer python-tomli python-pep517 python-setuptools python-flit-core python-wheel) pkgver=1 pkgrel=1 arch=(any) license=('custom') makedepends=(python git) url="https://gitlab.archlinux.org/archlinux/python-bootstrap" # From archlinux-contrib # package/parse-submodules https://github.com/jelly/python-bootstrap.git source=( "${pkgbase}::git+https://github.com/jelly/python-bootstrap.git#branch=3.11" "${pkgbase}-build::git+https://github.com/pypa/build.git" "${pkgbase}-flit::git+https://github.com/takluyver/flit.git" "${pkgbase}-installer::git+https://github.com/pypa/installer.git" "${pkgbase}-pep517::git+https://github.com/pypa/pep517.git" "${pkgbase}-setuptools::git+https://github.com/pypa/setuptools.git" "${pkgbase}-tomli::git+https://github.com/hukkin/tomli.git" "${pkgbase}-wheel::git+https://github.com/pypa/wheel.git" ) sha256sums=('SKIP' 'SKIP' 'SKIP' 'SKIP' 'SKIP' 'SKIP' 'SKIP' 'SKIP') prepare() { cd python-bootstrap git submodule init git config submodule."external/build".url "${srcdir}/${pkgbase}"-build git config submodule."external/flit".url "${srcdir}/${pkgbase}"-flit git config submodule."external/installer".url "${srcdir}/${pkgbase}"-installer git config submodule."external/pep517".url "${srcdir}/${pkgbase}"-pep517 git config submodule."external/setuptools".url "${srcdir}/${pkgbase}"-setuptools git config submodule."external/tomli".url "${srcdir}/${pkgbase}"-tomli git config submodule."external/wheel".url "${srcdir}/${pkgbase}"-wheel git -c protocol.file.allow=always submodule update git submodule update --init --recursive } build() { cd python-bootstrap python -m bootstrap.build } package_python-build() { pkgdesc="A simple, correct PEP 517 build frontend" cd python-bootstrap python -m bootstrap.install dist/build-*-py3-none-any.whl -d $pkgdir } package_python-installer() { pkgdesc="Low-level library for installing a Python package from a wheel distribution" cd python-bootstrap python -m bootstrap.install dist/installer-*-py3-none-any.whl -d $pkgdir } package_python-flit-core() { pkgdesc="Simplified packaging of Python modules (core backend)" cd python-bootstrap python -m bootstrap.install dist/flit_core-*-py3-none-any.whl -d $pkgdir } package_python-pep517() { pkgdesc="Wrappers to build Python packages using PEP 517 hooks" cd python-bootstrap python -m bootstrap.install dist/pep517-*-py3-none-any.whl -d $pkgdir } package_python-setuptools() { pkgdesc="Easily download, build, install, upgrade, and uninstall Python packages" cd python-bootstrap python -m bootstrap.install dist/setuptools-*-py3-none-any.whl -d $pkgdir } package_python-tomli() { pkgdesc="A lil' TOML parser" cd python-bootstrap python -m bootstrap.install dist/tomli-*-py3-none-any.whl -d $pkgdir } package_python-wheel() { pkgdesc="A built-package format for Python" cd python-bootstrap python -m bootstrap.install dist/wheel-*-py2.py3-none-any.whl -d $pkgdir } ``` ``` cp /usr/share/devtools/pacman-staging.conf /usr/share/devtools/pacman-python.conf ``` Edit the config file and add above [staging] ``` [python] SigLevel = Optional Server = https://pkgbuild.com/~jelle/python3.11 ``` ``` sudo ln -s /usr/bin/archbuild /usr/bin/python-x86_64-build ``` ``` repo-add python.db.tar.gz *.pkg.tar.zst ``` Trying python-tomli ``` ==> Starting build()... * Getting build dependencies for wheel... Traceback (most recent call last): File "/usr/lib/python3.11/site-packages/build/__main__.py", line 373, in main built = build_call( ^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/build/__main__.py", line 206, in build_package out = _build(isolation, builder, outdir, distribution, config_settings, skip_dependency_check) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/build/__main__.py", line 145, in _build return _build_in_current_env(builder, outdir, distribution, config_settings, skip_dependency_check) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/build/__main__.py", line 125, in _build_in_current_env missing = builder.check_dependencies(distribution) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/build/__init__.py", line 396, in check_dependencies return {u for d in dependencies for u in check_dependency(d)} ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/build/__init__.py", line 396, in <setcomp> return {u for d in dependencies for u in check_dependency(d)} ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/build/__init__.py", line 171, in check_dependency import packaging.requirements ModuleNotFoundError: No module named 'packaging' ERROR No module named 'packaging' ==> ERROR: A failure occurred in build(). Aborting... ==> ERROR: Build failed, check /var/lib/archbuild/python-x86_64/jelle/build [jelle@natrium][/tmp/python-tomli/trunk]% ``` The fix: ``` Index: PKGBUILD =================================================================== --- PKGBUILD (revision 459653) +++ PKGBUILD (working copy) @@ -8,13 +8,13 @@ license=('MIT') arch=('any') depends=('python') -makedepends=('python-build' 'python-installer' 'python-flit-core') +makedepends=('python-build' 'python-installer' 'python-flit-core' 'python-pep517') source=("https://github.com/hukkin/tomli/archive/$pkgver/$pkgname-$pkgver.tar.gz") sha512sums=('a467f8d48cdbd7213bd9b6f85fd48ba142ab7c9656c40bb30785e1c4b37a9e29eaed420f183458ad20112baee8413ebbec87755332795c8f02235d1018c3aa5c') build() { cd tomli-$pkgver - python -m build -wn + python -m build -wn --skip-dependency-check } check() { ``` python-flit ``` -> Extracting python-flit-3.7.1.tar.gz with bsdtar ==> Starting prepare()... Building sdist dist/flit_core-3.7.1.tar.gz Building wheel dist/flit_core-3.7.1-py3-none-any.whl ==> Starting build()... Traceback (most recent call last): File "<frozen runpy>", line 189, in _run_module_as_main File "<frozen runpy>", line 148, in _get_module_details File "<frozen runpy>", line 112, in _get_module_details File "/build/python-flit/src/flit-3.7.1/flit/__init__.py", line 12, in <module> from .config import ConfigError File "/build/python-flit/src/flit-3.7.1/flit/config.py", line 5, in <module> from .validate import validate_config File "/build/python-flit/src/flit-3.7.1/flit/validate.py", line 9, in <module> import requests ModuleNotFoundError: No module named 'requests' ==> ERROR: A failure occurred in build(). Aborting... ==> ERROR: Build failed, check /var/lib/archbuild/python-x86_64/jelle/build ``` ### TODO * derive list of initial bootstrap packages * check if we can provide a split package of for example python-build with a lower version number in [staging] * Create tooling to figure out the make/depends of the bootstrap * make de-vendoring of python-pdm-pep517 more easily switchable * move [python-bootstrap](https://github.com/FFY00/python-bootstrap) to our gitlab