On 30/01/2024 17:23, Vegard Nossum wrote:

We should still fix the writing of .rst to $(srctree), though -- our
use of parse-headers.pl seems to sidestep this by writing the 
intermediate .rst output to Documentation/output/, I'll have to look
a bit more closely.

I have now spent quite a bit of time investigating this.

The main result is that Sphinx really does NOT like it when documents
are located outside of the source root.

What makes the parse-headers.pl approach work is the 'kernel-include'
custom directive (from kernel_include.py) -- this allows you to include
files from outside the source root into a file that _is_ in the source
root. So the .rst files generated by parse-headers.pl are included using
this directive, rather than being treated as real documents that you can
refer to by their name.

If we did the same, one option would be to create one .rst file for each
.yaml file currently in Documentation/netlink/specs/ and just have them
'kernel-include' the corresponding generated file. I don't think this is
a great idea because we'd always have to remember to create new files
when a corresponding .yaml file is added -- and it feels like an ugly
workaround since each file would just be a single line including the
generated file.

Sphinx 7.2.0 introduces a mapping of document names (like
"networking/index") to paths, and it looks doable to write an extension
that would discover additional .rst files from outside the source path
to process. But before this there is essentially no way to have files
outside the source root appear in the toctree.

But I wouldn't like to bump the minimum required version from 2.4 to 7.2
either. Also, the extension wouldn't be using a public/standard API so
it would essentially be unsupported and could break again at any time.
It might not even be possible at all even today.

Another idea would be to have another layer of directories in the output
directory, so e.g.:

   $(BUILDDIR)/Documentation -> $(srctree)/Documentation (symlink)
   $(BUILDDIR)/generated/...

Of course this would break ALL the web links like

   https://docs.kernel.org/process/stable-kernel-rules.html

since it would now be

   https://docs.kernel.org/Documentation/process/stable-kernel-rules.html

so I consider this a no-go as well. (We could consider creating N
symlinks instead; one for each directory under Documentation/ -- but
this again goes well into hacky territory IMHO.)

As far as I can tell, the most viable remaining option is to use a true
Sphinx extension that defines a new directive (that points directly to a
YAML file) and never write out the .rst to a file.

I think this was Jani's preferred solution as well.

Jon, Breno -- what do you think?


Vegard