Re: [PATCH 07/13] docs: move sphinx-pre-install to tools/doc

Em Fri, 15 Aug 2025 07:18:51 -0600
Jonathan Corbet [off-list ref] escreveu:

Mauro Carvalho Chehab [off-list ref] writes:

quoted

quoted

Between "tools/doc" and "tools/docs" I don't really have overly strong
feelings; if you work has the latter we can just stick with that.  If
you propose "tools/Documentation", though, expect resistance :)  

<joke>
Heh, I'm tempted to propose:
	/Documentation -> /docs
or
	/Documentation -> /Docs
</joke>  

I proposed the former at a maintainers summit a few years ago ... the
response was less than fully positive.  I guess the fact that docs have
the longest and only capitalized top-level directory name shows their
relative importance :)

:-)

quoted

Ok, so let's keep tools/docs then. We need to decide about python
lib. On my series, I'm placing at tools/docs/lib, but I guess we
can change it later.

From my side, I would prefer to have a single directory for tools,
as we may place there things that aren't specific to docs.

For instance, I have my own class that I use for command execution,
using asyncio. The rationale is that it allows output messages in
real time without needing to wait for the entire process to end(*).

(*) I recently discovered a way to do that without needing asyncio,
    which makes the code a little bit simpler.  

This is just flushing the output buffer?  Asyncio seems like a heavy
tool for that; I guess I'm missing something.

Yes, flushing output as they arrive while storing results from stdout and
stderr and handling  "stdin", eventually dynamically printing stdin when
debug is enabled. The problem sounds simple, and this is really easy to
implement in Perl. However, on Python, this ends to be a very complex
problem that one can only get all the caveats after implementing unit 
tests and then discovering, for instance, that, if stdin is bigger than 
32KB, Python read logic freezes forever (or until gather() timeout, and
the stdin was incomplete.

To fix it, one needs to implement a different stdin function that can't
wait anymore for the end of a read operation, but instead needs to pick
buffers in small chunks. Plus, the subprocess execution method requires
some unusual buffering parameters.
 

quoted

Either using asyncio or not, a class for such purpose is something
that multiple tools could use. So, a generic dir like tools/lib, 
lib/python, ... IMO makes sense.  

I agree with this, anyway.  "tools/python" might end up as the winning
suggestion. 

quoted

quoted

As I said, my series was an RFC to see what it would look like; it did't
take all that long the first time around, and will be even quicker to
redo on top of a new base, whatever that turns out to be.  

With regards to the RFC, IMO we still may need to discuss how we'll end 
placing libraries under a LIBDIR. IMO, your RFC should also propose
a directory structure. I mean, we could have something like:

	LIBDIR     # either tools/docs/lib, tools/lib, lib/python or whatever
	|
	+---> common
	\---> docs
		|
	    	+---> kdoc
	    	\---> abi

We could instead do:
	- flatten "common" to LIBDIR; or:
	- flatten "docs" to LIBDIR; or:
	- flatten both but keeping kdoc, abi, ... directories inside
	  LIBDIR; or:
	- have a completely flatten directory with everything
	  under LIBDIR.  

I'm not sure we need the common/docs intermediate directory.

Meanwhile, I had a related, possibly unpopular idea...  Start with
.../tools/python/kernel and put a basic __init__.py file there;
everything else would go into that directory or before.  The imports
would then read something like:

  from kernel import abi_parser

Not against something similar to it, but IMO "kernel" is a bad
name as it sounds something that runs in kernel stace or for Kernel
build. It could be, instead:

	from lib import abi_parser

Yet, I guess it may still need to add something at PATH, depending from
where current working dir the script was called (but tests required).

For instance Documentation/sphinx would need to pick lib from
"../../tools/python". So, I guess Sphinx extensions need first to
do:

	sys.path.insert(0, "../../tools/python")

and then:

	from lib import abi_parser         # or:
	from lib.abi import abi_parser     # or:
	import .abi_parser


Btw, nothing prevents moving extensions from Documentation/sphinx
into tools/sphinx_extensions. We just need to add the path insert
at conf.py.

That would make it clear which modules are ours and which come from
elsewhere.

I was planning to toss together another RFC with that scheme, again to
see what it would look like in practice.

Thoughts?

Fine from my side.

Regards,
Mauro