Since the release of Docutils 0.18 today building Python documentation fails.

In CPython repository we have Sphinx pinned to 3.2.1, which has too loose requirement for Docutils version in install_requires – docutils>=0.12.

Example of failing build: https://github.com/python/python-docs-pl/runs/4009459257.

My suggestion is to pin down docutils in docs requirements.

The pin can be removed after bumping Sphinx to at least 3.5.4, which introduced upper limit for docutils version.

https://github.com/sphinx-doc/sphinx/commit/9263eea38379822e053c67ad1d17036d724c5e41

New changeset bcee6aa31550cfecdc3acecbd0e4447bb0051887 by m-aciek in branch 'main':
bpo-45618: Fix documentation build by pinning Docutils version to 0.17.1 (GH-29230)
https://github.com/python/cpython/commit/bcee6aa31550cfecdc3acecbd0e4447bb0051887

New changeset 2b7b7c7320cd856df5439afc3c984873678c27d8 by Miss Islington (bot) in branch '3.10':
bpo-45618: Fix documentation build by pinning Docutils version to 0.17.1 (GH-29230) (GH-29240)
https://github.com/python/cpython/commit/2b7b7c7320cd856df5439afc3c984873678c27d8

New changeset 30c1f18ee62cef301e18488b80b4d329290f7b95 by Ned Deily in branch '3.9':
[3.9] bpo-45618: Fix documentation build by pinning Docutils version to 0.17.1 (GH-29230) (GH-29241)
https://github.com/python/cpython/commit/30c1f18ee62cef301e18488b80b4d329290f7b95

All of the open branches are affected by this since we also have to build documentation for security-fix-only branches when releases. PRs for 3.7 and 3.6 forthcoming.

For what it's worth I have just successfully built documentation on 3.7 and 3.6 branches locally.  It looks like Sphinx 2.3.1 used there doesn't use this part of docutils API that have changed recently.

Sphinx 2.3.1 has requirement of "docutils>=0.12", just like 3.2.1.

We can pin docutils version to current (0.18) "just in case" to prevent a breakage in the future or do nothing – maybe it won't break – maybe the docutils API used is narrow enough.

It occurs that the documentation builds without warnings on most recent Sphinx -- version 4.2.0. Probably we should bump the version in 3.11, 3.10 and 3.9 to 4.2.0. That would enable us to remove the pin on docutils.

I'm sorry for the confusion. I might have make more checking on my side before proposing a fix.

New changeset f2407144347fafcd69c2ade41b5d9c3fb07b59ef by Ned Deily in branch '3.8':
[3.9] bpo-45618: Fix documentation build by pinning Docutils version to 0.17.1 (GH-29230) (GH-29241) (GH-29245)
https://github.com/python/cpython/commit/f2407144347fafcd69c2ade41b5d9c3fb07b59ef

New changeset 14a4fce457033412278ca9a056787db424310dc3 by m-aciek in branch 'main':
bpo-45618: Update Sphinx version used to build the documentation to 4.2.0 (GH-29256)
https://github.com/python/cpython/commit/14a4fce457033412278ca9a056787db424310dc3

New changeset 7a5850987010ca5cb0f1e2844cfe183935916e7e by Ned Deily in branch '3.7':
bpo-45618: Fix documentation build by pinning Docutils version to 0.17.1 (GH-31476)
https://github.com/python/cpython/commit/7a5850987010ca5cb0f1e2844cfe183935916e7e

After the Jinja version 3.1.0 release today [1], documentation builds fail for all stable branches. Sphinx in versions pinned on those branches fails with ImportError. [2]

Backporting 14a4fce457033412278ca9a056787db424310dc3 to 3.10 and 3.9 fixes the problem for them (build is successful with 4.2.0). For 3.8 and 3.7 pinning down Jinja is required (docs don't build smoothly with 4.2.0 for those branches).

Could some core developer help with the backports?

I am starting to work on PRs with pins for 3.8 and 3.7.


[1] https://jinja.palletsprojects.com/en/3.1.x/changes/#version-3-1-0
[2] % sphinx-build --version                                  
Traceback (most recent call last):
  File "…/cpython/Doc/./venv/bin/sphinx-build", line 5, in <module>
    from sphinx.cmd.build import main
  File "…/cpython/Doc/venv/lib/python3.9/site-packages/sphinx/cmd/build.py", line 25, in <module>
    from sphinx.application import Sphinx
  File "…/cpython/Doc/venv/lib/python3.9/site-packages/sphinx/application.py", line 42, in <module>
    from sphinx.registry import SphinxComponentRegistry
  File "…/cpython/Doc/venv/lib/python3.9/site-packages/sphinx/registry.py", line 24, in <module>
    from sphinx.builders import Builder
  File "…/cpython/Doc/venv/lib/python3.9/site-packages/sphinx/builders/__init__.py", line 26, in <module>
    from sphinx.util import import_object, logging, rst, progress_message, status_iterator
  File "…/cpython/Doc/venv/lib/python3.9/site-packages/sphinx/util/rst.py", line 22, in <module>
    from jinja2 import environmentfilter
ImportError: cannot import name 'environmentfilter' from 'jinja2' (…/cpython/Doc/venv/lib/python3.9/site-packages/jinja2/__init__.py)

This really should be handled in a new issue since the original fixes for all affected releases are already in the field. Moreover, we should be a bit careful about bumping the required Sphinx version for older releases as this has caused problems for downstream distributors of Python in the past. Can't we just pin the Jinja2 version for current releases?

> This really should be handled in a new issue since the original fixes for all affected releases are already in the field.

Sorry, I got suggested too much with expected fix which is common for both of those issues. Shall I create a new issue?

> Moreover, we should be a bit careful about bumping the required Sphinx version for older releases as this has caused problems for downstream distributors of Python in the past. Can't we just pin the Jinja2 version for current releases?

Didn't know about the problems for distributors. Definitely just pinning Jinja2 is enough.

> > This really should be handled in a new issue since the original fixes for all affected releases are already in the field.

> Sorry, I got suggested too much with expected fix which is common for both of those issues. Shall I create a new issue?

I created https://bugs.python.org/issue47138, let's continue there.