332 views
owned this note
# Bootstrapping Python
2022-10-31
Attendees:
* felixonmars
* FFY00
* dvzrv
* jelle
## Agenda
Some of the topics have been highlighted in https://wiki.archlinux.org/title/User:Davezerave/Python_Bootstrapping
### Building Python 3.11
Already build 3.11 and bootstrapped packages
https://pkgbuild.com/~jelle/python3.11/
### Bootstrap Python 3.11
For bootstrapping the basic packages we currently need:
* python-build
* [python-flit-core](https://github.com/pypa/flit/tree/main/flit_core) ([self-bootstrapping](https://github.com/pypa/flit/blob/7b13fb2d1425411cc0c03bb9ed9d49264c1e4dbb/flit_core/pyproject.toml#L1-L4)) (python-tomli or backport https://github.com/pypa/flit/commit/dba0f317c52d3c9fa29792a0beb0c0abf4476a23)
* python-setuptools (ensurepip hack: https://github.com/archlinux/svntogit-packages/commit/8ac0104aa4758689c2aaf938a1e2043466cf3d03)
* python-installer
* [python-poetry-core](https://github.com/python-poetry/poetry-core) ([self-bootstrapping](https://github.com/python-poetry/poetry-core/blob/dd1245859e8a2349d26d66a0ebdcb82ff3fc09db/pyproject.toml#L118-L121))
* [python-pdm-pep517](https://github.com/pdm-project/pdm-pep517) ([self-bootstrapping](https://github.com/pdm-project/pdm-pep517/blob/cb6eefedb163eae76dd4ba9392608585fe9193b1/pyproject.toml#L52-L55) if vendored!)
* [python-hatch](https://github.com/pypa/hatch/) ([self-bootstrapping](https://github.com/pypa/hatch/blob/master/backend/pyproject.toml))
(there may be hidden dependencies, reported only via the runtime `get_requires_for_build_wheel` hook)
bootstrap currently provides:
* build
* flit_core
* installer
* pep517
* setuptools
* tomli
* wheel
### Bootstrap build backends
Some build backends may require bootstraping themselves, if one of their dependencies use said backend to build.
#### List of build backends in our repos:
- python-setuptools
- python-flit-core + python-flit
- python-poetry-core + python-poetry
- python-pdm-pep517 + python-pdm
- python-hatchling + python-hatch
- python-sphinx-theme-builder
- meson-python
### Bootstrap workflow suggestion
* Use python-bootstrap to create each(?!) bootstrap package (e.g. python-build-bootstrap providing python-build, etc.) in [staging]
* `python -m bootstrap.install $wheel --destdir "$pkgdir"`
* Build all build and runtime requirements of the initial bootstrap packages
* Build all initial bootstrap packages properly and replace the initial bootstrap packages
* Commence with rebuild as normal
https://github.com/jelly/python-bootstrap/tree/PKGBUILD
```
# Maintainer
pkgbase=python-bootstrap
pkgname=(python-build python-installer python-tomli python-pep517 python-setuptools python-flit-core python-wheel)
pkgver=1
pkgrel=1
arch=(any)
license=('custom')
makedepends=(python git)
url="https://gitlab.archlinux.org/archlinux/python-bootstrap"
# From archlinux-contrib
# package/parse-submodules https://github.com/jelly/python-bootstrap.git
source=(
"${pkgbase}::git+https://github.com/jelly/python-bootstrap.git#branch=3.11"
"${pkgbase}-build::git+https://github.com/pypa/build.git"
"${pkgbase}-flit::git+https://github.com/takluyver/flit.git"
"${pkgbase}-installer::git+https://github.com/pypa/installer.git"
"${pkgbase}-pep517::git+https://github.com/pypa/pep517.git"
"${pkgbase}-setuptools::git+https://github.com/pypa/setuptools.git"
"${pkgbase}-tomli::git+https://github.com/hukkin/tomli.git"
"${pkgbase}-wheel::git+https://github.com/pypa/wheel.git"
)
sha256sums=('SKIP'
'SKIP'
'SKIP'
'SKIP'
'SKIP'
'SKIP'
'SKIP'
'SKIP')
prepare() {
cd python-bootstrap
git submodule init
git config submodule."external/build".url "${srcdir}/${pkgbase}"-build
git config submodule."external/flit".url "${srcdir}/${pkgbase}"-flit
git config submodule."external/installer".url "${srcdir}/${pkgbase}"-installer
git config submodule."external/pep517".url "${srcdir}/${pkgbase}"-pep517
git config submodule."external/setuptools".url "${srcdir}/${pkgbase}"-setuptools
git config submodule."external/tomli".url "${srcdir}/${pkgbase}"-tomli
git config submodule."external/wheel".url "${srcdir}/${pkgbase}"-wheel
git -c protocol.file.allow=always submodule update
git submodule update --init --recursive
}
build() {
cd python-bootstrap
python -m bootstrap.build
}
package_python-build() {
pkgdesc="A simple, correct PEP 517 build frontend"
cd python-bootstrap
python -m bootstrap.install dist/build-*-py3-none-any.whl -d $pkgdir
}
package_python-installer() {
pkgdesc="Low-level library for installing a Python package from a wheel distribution"
cd python-bootstrap
python -m bootstrap.install dist/installer-*-py3-none-any.whl -d $pkgdir
}
package_python-flit-core() {
pkgdesc="Simplified packaging of Python modules (core backend)"
cd python-bootstrap
python -m bootstrap.install dist/flit_core-*-py3-none-any.whl -d $pkgdir
}
package_python-pep517() {
pkgdesc="Wrappers to build Python packages using PEP 517 hooks"
cd python-bootstrap
python -m bootstrap.install dist/pep517-*-py3-none-any.whl -d $pkgdir
}
package_python-setuptools() {
pkgdesc="Easily download, build, install, upgrade, and uninstall Python packages"
cd python-bootstrap
python -m bootstrap.install dist/setuptools-*-py3-none-any.whl -d $pkgdir
}
package_python-tomli() {
pkgdesc="A lil' TOML parser"
cd python-bootstrap
python -m bootstrap.install dist/tomli-*-py3-none-any.whl -d $pkgdir
}
package_python-wheel() {
pkgdesc="A built-package format for Python"
cd python-bootstrap
python -m bootstrap.install dist/wheel-*-py2.py3-none-any.whl -d $pkgdir
}
```
```
cp /usr/share/devtools/pacman-staging.conf /usr/share/devtools/pacman-python.conf
```
Edit the config file and add above [staging]
```
[python]
SigLevel = Optional
Server = https://pkgbuild.com/~jelle/python3.11
```
```
sudo ln -s /usr/bin/archbuild /usr/bin/python-x86_64-build
```
```
repo-add python.db.tar.gz *.pkg.tar.zst
```
Trying python-tomli
```
==> Starting build()...
* Getting build dependencies for wheel...
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/build/__main__.py", line 373, in main
built = build_call(
^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/build/__main__.py", line 206, in build_package
out = _build(isolation, builder, outdir, distribution, config_settings, skip_dependency_check)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/build/__main__.py", line 145, in _build
return _build_in_current_env(builder, outdir, distribution, config_settings, skip_dependency_check)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/build/__main__.py", line 125, in _build_in_current_env
missing = builder.check_dependencies(distribution)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/build/__init__.py", line 396, in check_dependencies
return {u for d in dependencies for u in check_dependency(d)}
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/build/__init__.py", line 396, in <setcomp>
return {u for d in dependencies for u in check_dependency(d)}
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/build/__init__.py", line 171, in check_dependency
import packaging.requirements
ModuleNotFoundError: No module named 'packaging'
ERROR No module named 'packaging'
==> ERROR: A failure occurred in build().
Aborting...
==> ERROR: Build failed, check /var/lib/archbuild/python-x86_64/jelle/build
[jelle@natrium][/tmp/python-tomli/trunk]%
```
The fix:
```
Index: PKGBUILD
===================================================================
--- PKGBUILD (revision 459653)
+++ PKGBUILD (working copy)
@@ -8,13 +8,13 @@
license=('MIT')
arch=('any')
depends=('python')
-makedepends=('python-build' 'python-installer' 'python-flit-core')
+makedepends=('python-build' 'python-installer' 'python-flit-core' 'python-pep517')
source=("https://github.com/hukkin/tomli/archive/$pkgver/$pkgname-$pkgver.tar.gz")
sha512sums=('a467f8d48cdbd7213bd9b6f85fd48ba142ab7c9656c40bb30785e1c4b37a9e29eaed420f183458ad20112baee8413ebbec87755332795c8f02235d1018c3aa5c')
build() {
cd tomli-$pkgver
- python -m build -wn
+ python -m build -wn --skip-dependency-check
}
check() {
```
python-flit
```
-> Extracting python-flit-3.7.1.tar.gz with bsdtar
==> Starting prepare()...
Building sdist
dist/flit_core-3.7.1.tar.gz
Building wheel
dist/flit_core-3.7.1-py3-none-any.whl
==> Starting build()...
Traceback (most recent call last):
File "<frozen runpy>", line 189, in _run_module_as_main
File "<frozen runpy>", line 148, in _get_module_details
File "<frozen runpy>", line 112, in _get_module_details
File "/build/python-flit/src/flit-3.7.1/flit/__init__.py", line 12, in <module>
from .config import ConfigError
File "/build/python-flit/src/flit-3.7.1/flit/config.py", line 5, in <module>
from .validate import validate_config
File "/build/python-flit/src/flit-3.7.1/flit/validate.py", line 9, in <module>
import requests
ModuleNotFoundError: No module named 'requests'
==> ERROR: A failure occurred in build().
Aborting...
==> ERROR: Build failed, check /var/lib/archbuild/python-x86_64/jelle/build
```
### TODO
* derive list of initial bootstrap packages
* check if we can provide a split package of for example python-build with a lower version number in [staging]
* Create tooling to figure out the make/depends of the bootstrap
* make de-vendoring of python-pdm-pep517 more easily switchable
* move [python-bootstrap](https://github.com/FFY00/python-bootstrap) to our gitlab