On Wed, 10 Sep 2025, Mauro Carvalho Chehab [off-list ref] wrote:

Em Wed, 10 Sep 2025 13:46:17 +0300
Jani Nikula [off-list ref] escreveu:

quoted

On Thu, 04 Sep 2025, Mauro Carvalho Chehab [off-list ref] wrote:

quoted

There are too much magic inside docs Makefile to properly run
sphinx-build. Create an ancillary script that contains all
kernel-related sphinx-build call logic currently at Makefile.

Such script is designed to work both as an standalone command
and as part of a Makefile. As such, it properly handles POSIX
jobserver used by GNU make.

On a side note, there was a line number increase due to the
conversion:

 Documentation/Makefile          |  131 +++----------
 tools/docs/sphinx-build-wrapper |  293 +++++++++++++++++++++++++++++++
 2 files changed, 323 insertions(+), 101 deletions(-)

This is because some things are more verbosed on Python and because
it requires reading env vars from Makefile. Besides it, this script
has some extra features that don't exist at the Makefile:

- It can be called directly from command line;
- It properly return PDF build errors.

When running the script alone, it will only take handle sphinx-build
targets. On other words, it won't runn make rustdoc after building
htmlfiles, nor it will run the extra check scripts.  

I've always strongly believed we should aim to make it possible to build
the documentation by running sphinx-build directly on the
command-line. Not that it would be the common way to run it, but to not
accumulate things in the Makefile that need to happen before or
after. To promote handling the documentation build in Sphinx. To be able
to debug issues and try new Sphinx versions without all the hacks.

That would be the better, but, unfortunately, this is not possible, for 
several reasons:

1. SPHINXDIRS. It needs a lot of magic to work, both before running
   sphinx-build and after (inside conf.py);

Makes you wonder if that's the right solution to the original
problem. It was added as a kind of hack, and it stuck.

2. Several extensions require kernel-specific environment variables to
   work. Calling sphinx-build directly breaks them;

The extensions shouldn't be using environment variables for
configuration anyway. Add config options and set them in conf.py like
everything else?

3. Sphinx itself doesn't build several targets alone. Instead, they create
   a Makefile, and an extra step is needed to finish the build. That's 
   the case for pdf and texinfo, for instance;

That's not true for the Makefile currently generated by
sphinx-quickstart. Granted, I haven't used Sphinx much for pdf output.

4. Man pages generation. Sphinx support to generate it is very poor;

In what way?

5. Rust integration adds more complexity to the table;

I'm not seeing sphinx-build supporting the above needs anytime soon,
and, even if we push our needs to Sphinx and it gets accepted there,
we'll still need to wait for quite a while until LTS distros merge
them.

I'm not suggesting to add anything to Sphinx upstream.

quoted

This patch moves a bunch of that logic into a Python wrapper, and I feel
like it complicates matters. You can no longer rely on 'make V=1' to get
the build commands, for instance.

Quite the opposite. if you try using "make V=1", it won't show the
command line used to call sphinx-build anymore.

This series restore it.

See, if you build with this series with V=1, you will see exactly
what commands are used on the build:

	$ make V=1 htmldocs
	...
	python3 ./tools/docs/sphinx-build-wrapper htmldocs \
	        --sphinxdirs="." --conf="conf.py" \
        	--builddir="Documentation/output" \
	        --theme= --css= --paper=
	python3 /new_devel/docs/sphinx_latest/bin/sphinx-build -j25 -b html -c /new_devel/docs/Documentation -d /new_devel/docs/Documentation/output/.doctrees -D kerneldoc_bin=scripts/kernel-doc.py -D version=6.17.0-rc1 -D release=6.17.0-rc1+ -D kerneldoc_srctree=. /new_devel/docs/Documentation /new_devel/docs/Documentation/output
	...

quoted

Newer Sphinx versions have the -M option for "make mode". The Makefiles
produced by sphinx-quickstart only have one build target:

# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).

I didn't know about this, but from [1] it sounds it covers just two
targets: "latexpdf" and "info".

sphinx-build -M help gives a list of 24 targets.

The most complex scenario is still not covered: SPHINXDIRS.

[1] https://www.sphinx-doc.org/en/master/man/sphinx-build.html

quoted

%: Makefile
        @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

That's all.

Try doing such change on your makefile. it will break:

	- SPHINXDIRS;
	- V=1;
	- rustdoc

I know it does. That's the problem.

and will still be dependent on variables that are passed via
env from Kernel makefile. So, stil you can't run from command
line. Also, if you call sphinx-build from command line:

	$ sphinx-build -j25 -b html Documentation Documentation/output
	...
	      File "<frozen os>", line 717, in __getitem__
	    KeyError: 'srctree'

It won't work, as several parameters that are required by conf.py and by
Sphinx extensions would be missing (the most important one is srctree, but
there are others in the line too).

quoted

The proposed wrapper duplicates loads of code that's supposed to be
handled by sphinx-build directly.

Once we get the wrapper, we can work to simplify it, but still I
can't see how to get rid of it.

I just don't understand the mentality of first adding something complex,
and then working to simplify it.

Don't make it a Rube Goldberg machine in the first place.

quoted

Including the target/builder names.

True, but this was a design decision taken lots of years ago: instead
of:
	make html

we're using:

	make htmldocs

This series doesn't change that: either makefile or the script need
to tho the namespace conversion.

In the above Makefile snippet that conversion would be $(@:docs=)

The clean Makefile way of checking for having Sphinx and the required
versions of Python and dependencies etc. would be a .PHONY target that
just checks, and doesn't do *anything* else. It shouldn't be part of the
sphinx-build rules.

PHONY += check-versions
check-versions:
	sphinx-pre-install --version-check

htmldocs: check-versions
	...

Or something like that.

quoted

Seems to me the goal should be to figure out *generic* wrappers for
handling parallelism, not Sphinx aware/specific.


BR,
Jani.

-- 
Jani Nikula, Intel