Re: [PATCH 0/8] (REBASED) get_abi.pl undefined: improve precision and performance

On Mon, Sep 27, 2021 at 03:39:42PM +0200, Mauro Carvalho Chehab wrote:

Em Mon, 27 Sep 2021 11:23:20 +0200
Greg Kroah-Hartman [off-list ref] escreveu:

quoted

On Mon, Sep 27, 2021 at 10:55:53AM +0200, Mauro Carvalho Chehab wrote:

quoted

Em Thu, 23 Sep 2021 19:13:04 +0200
Greg Kroah-Hartman [off-list ref] escreveu:
  

quoted

On Thu, Sep 23, 2021 at 05:41:11PM +0200, Mauro Carvalho Chehab wrote:  

quoted

Hi Greg,

As requested, this is exactly the same changes, rebased on the top of
driver-core/driver-core-next.

-

It follows a series of improvements for get_abi.pl. it is on the top of driver-core/driver-core-next.

With such changes, on my development tree, the script is taking 6 seconds to run 
on my desktop:

	$ !1076
	$ time ./scripts/get_abi.pl undefined |sort >undefined_after && cat undefined_after| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined_after undefined_symbols

	real	0m6,292s
	user	0m5,640s
	sys	0m0,634s
	  6838 undefined_after
	   808 undefined_symbols
	  7646 total

And 7 seconds on a Dell Precision 5820:

	$ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols

	real	0m7.162s
	user	0m5.836s
	sys	0m1.329s
	6548 undefined
	772 undefined_symbols

Both tests were done against this tree (based on today's linux-next):

	$ https://git.kernel.org/pub/scm/linux/kernel/git/mchehab/devel.git/log/?h=get_abi_undefined-latest

It should be noticed that, as my tree has several ABI fixes,  the time to run the
script is likely less than if you run on your tree, as there will be less symbols to
be reported, and the algorithm is optimized to reduce the number of regexes
when a symbol is found.

Besides optimizing and improving the seek logic, this series also change the
debug logic. It how receives a bitmap, where "8" means to print the regexes
that will be used by "undefined" command:

	$ time ./scripts/get_abi.pl undefined --debug 8 >foo
	real	0m17,189s
	user	0m13,940s
	sys	0m2,404s

	$wc -l foo
	18421939 foo

	$ cat foo
	...
	/sys/kernel/kexec_crash_loaded =~ /^(?^:^/sys/.*/iio\:device.*/in_voltage.*_scale_available$)$/
	/sys/kernel/kexec_crash_loaded =~ /^(?^:^/sys/.*/iio\:device.*/out_voltage.*_scale_available$)$/
	/sys/kernel/kexec_crash_loaded =~ /^(?^:^/sys/.*/iio\:device.*/out_altvoltage.*_scale_available$)$/
	/sys/kernel/kexec_crash_loaded =~ /^(?^:^/sys/.*/iio\:device.*/in_pressure.*_scale_available$)$/
	...

On other words, on my desktop, the /sys match is performing >18M regular 
expression searches, which takes 6,2 seconds (or 17,2 seconds, if debug is 
enabled and sent to an area on my nvme storage).    

Better, it's down to 10 minutes on my machine now:

	real	10m39.218s
	user	10m37.742s
	sys	0m0.775s  

A lot better, but not clear why it is still taking ~40x more than here...
It could well be due to the other ABI changes yet to be applied
(I'll submit it probably later today), but it could also be related to
something else. Could this be due to disk writes?  

Disk writes to where for what?  This is a very fast disk (nvme raid
array)  It's also a very "big" system, with lots of sysfs files:

	$ find /sys/devices/ -type f | wc -l
	44334

Ok. Maybe that partially explains why it is taking so long, as the
number of regex to compare will increase (not linearly).

No idea.  I just ran it on my laptop and it took only 5 seconds.

Hm, you aren't reading the values of the sysfs files, right?

Anything I can do to run to help figure out where the script is taking
so long?

quoted

And any Documentation/ABI/ updates you might have I'll gladly
take as well.

I'll be submitting it soon enough. Got sidetracked by a regression
on my INBOX due to a fetchmail regression[1].

Ick, fetchmail.  I recommend getmail instead, much more robust and a
sane maintainer :)

I'll take a look at those patches now.

thanks,

greg k-h