Skip to content

COMP: Bootstrap pip via ensurepip on Windows wheel build to fix Python 3.10+#304

Open
hjmjohnson wants to merge 1 commit intomainfrom
fix-pip-collections-mapping
Open

COMP: Bootstrap pip via ensurepip on Windows wheel build to fix Python 3.10+#304
hjmjohnson wants to merge 1 commit intomainfrom
fix-pip-collections-mapping

Conversation

@hjmjohnson
Copy link
Copy Markdown
Member

@hjmjohnson hjmjohnson commented Apr 11, 2026

Bootstrap pip via python -m ensurepip --upgrade --default-pip before invoking the system pip in windows_build_module_wheels.py. Sidesteps a Python 3.10+ chicken-and-egg failure where the system pip's vendored html5lib does from collections import Mapping (removed in 3.10) and dies before any pip install can run.

Surfaced by KitwareMedical/ITKUltrasound#246.

Failure trace
File ".../Temp/tmpXXXX/pip.zip/pip/_vendor/html5lib/_trie/_base.py", line 3
ImportError: cannot import name 'Mapping' from 'collections' (C:\Python310-x64\lib\collections\__init__.py)

Consistent on Python 3.10 and 3.11 across all build-windows-python-packages matrix jobs.

Root cause

scripts/windows-download-cache-and-build-module-wheels.ps1 IEXes the upstream install-python.ps1 from scikit-build/scikit-ci-addons. That script's line 160 downloads get-pip.py from a pinned GitHub gist (jcfr/db7347e8708b9f32d45ab36125fad6d3@8478d43e) rather than https://bootstrap.pypa.io/get-pip.py. The pinned gist installs a pip old enough to vendor an html5lib that still uses from collections import Mapping.

When windows_build_module_wheels.py:55 runs python -m pip install pip --upgrade, the system pip loads its _vendor/html5lib._trie._base and dies before any package-resolution logic runs. The bug cannot be worked around using the broken pip itself.

Why ensurepip works

ensurepip installs the pip wheel that CPython itself bundled at release time (Lib/ensurepip/_bundled/), which is ≥ 21.x for Python 3.10 and ≥ 23.x for 3.11 — both ship a working html5lib ≥ 1.1 with the collections.abc.Mapping migration applied. Once ensurepip writes a fresh pip into the system Python, the subsequent pip install pip --upgrade finds a working pip and proceeds normally.

What is NOT changed
  • windows-download-cache-and-build-module-wheels.ps1: still IEXes the upstream install-python.ps1 unchanged. The fix is purely Python-side; no scikit-ci-addons interaction needed.
  • scripts/windows_build_wheels.py: unchanged. Its pip operations go through a virtualenv created by virtualenv.exe, and modern virtualenv ≥ 20.x ships its own bundled pip seed wheels instead of copying the (broken) system pip into the new venv.
  • The upstream scikit-ci-addons pinned gist URL: ideally fixed at the source eventually (replace with bootstrap.pypa.io/get-pip.py), but that's a separate, broader-scope change in a different repo.

Test plan

…n 3.10+

scikit-ci-addons' install-python.ps1 downloads get-pip.py from a pinned
GitHub gist (jcfr commit 8478d43e) rather than the canonical
bootstrap.pypa.io URL, and installs a pip old enough to vendor an
html5lib that still uses `from collections import Mapping`.  That
import was removed in Python 3.10, so the very first invocation of
`python -m pip install pip --upgrade` (the bootstrap step in
`build_wheels`) dies with:

    File ".../pip.zip/pip/_vendor/html5lib/_trie/_base.py", line 3
    ImportError: cannot import name 'Mapping' from 'collections'

The failure is consistent on Python 3.10 and 3.11 across all Windows
matrix jobs (build-windows-python-packages (10) and (11)).

Fix it by running `python -m ensurepip --upgrade --default-pip` BEFORE
the existing `pip install pip --upgrade`.  ensurepip uses the pip wheel
that CPython itself bundled at release time (>= 21.x for Python 3.10),
which has a working html5lib >= 1.1, and the subsequent self-upgrade
then proceeds normally.

The PowerShell entry point (windows-download-cache-and-build-module-wheels.ps1)
still IEXes the upstream install-python.ps1 unchanged - the chicken-and-egg
fix is purely Python-side, so no interaction with scikit-ci-addons is
required.  Companion `windows_build_wheels.py` is unchanged because its
pip operations go through a virtualenv created by `virtualenv.exe`, and
modern virtualenv ships its own bundled pip wheels rather than copying
the (broken) system pip.

Surfaced by KitwareMedical/ITKUltrasound#246, which is the first remote
module PR to actually exercise the python wheel build matrix on Python
3.10+ after a related CI fix made the workflow start running again.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant