[PATCH v3 0/7] get_abi.pl: Check for missing symbols at the ABI specs

STALE1776d

Revision v3 of 3 in this series.

5 messages, 2 authors, 2021-09-22 · open the first message on its own page

[PATCH v3 0/7] get_abi.pl: Check for missing symbols at the ABI specs

From: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Date: 2021-09-18 09:52:48

Hi Greg,

Add a new feature at get_abi.pl to optionally check for existing symbols
under /sys that won't match a "What:" inside Documentation/ABI.

Such feature is very useful to detect missing documentation for ABI.

This series brings a major speedup, plus it fixes a few border cases when
matching regexes that end with a ".*" or \d+.

patch 1 changes get_abi.pl logic to handle multiple What: lines, in
order to make the script more robust;

patch 2 adds the basic logic. It runs really quicky (up to 2
seconds), but it doesn't use sysfs softlinks.

Patch 3 adds support for parsing softlinks. It makes the script a
lot slower, making it take a couple of minutes to process the entire
sysfs files. It could be optimized in the future by using a graph,
but, for now, let's keep it simple.

Patch 4 adds an optional parameter to allow filtering the results
using a regex given by the user. When this parameter is used
(which should be the normal usecase), it will only try to find softlinks
if the sysfs node matches a regex.

Patch 5 improves the report by avoiding it to ignore What: that
ends with a wildcard.

Patch 6 is a minor speedup.  On a Dell Precision 5820, after patch 6, 
results are:

	$ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols

	real	2m35.563s
	user	2m34.346s
	sys	0m1.220s
	7595 undefined
	896 undefined_symbols

Patch 7 makes a *huge* speedup: it basically switches a linear O(n^3)
search for links by a logic which handle symlinks using BFS. It
also addresses a border case that was making 'msi-irqs/\d+' regex to
be misparsed. 

After patch 7, it is 11 times faster:

	$ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols

	real	0m14.137s
	user	0m12.795s
	sys	0m1.348s
	7030 undefined
	794 undefined_symbols

(the difference on the number of undefined symbols are due to the fix for
it to properly handle 'msi-irqs/\d+' regex)

-

While this series is independent from Documentation/ABI changes, it
works best when applied from this tree, which also contain ABI fixes
and a couple of additions of frequent missed symbols on my machine:

    https://git.kernel.org/pub/scm/linux/kernel/git/mchehab/devel.git/log/?h=get_undefined_abi_v3

-

v3:
  - Fixed parse issues with 'msi-irqs/\d+' regex;
  - Added a BFS graph logic to solve symlinks at sysfs;

v2:
  - multiple What: for the same description are now properly handled;
  - some special cases are now better handled;
  - some bugs got fixed.

The full series, with the ABI changes and some ABI improvements can be found
at:
	https://git.kernel.org/pub/scm/linux/kernel/git/mchehab/devel.git/commit/?h=get_undefined&id=1838d8fb149170f6c19feda0645d6c3157f46f4f



Mauro Carvalho Chehab (7):
  scripts: get_abi.pl: Better handle multiple What parameters
  scripts: get_abi.pl: Check for missing symbols at the ABI specs
  scripts: get_abi.pl: detect softlinks
  scripts: get_abi.pl: add an option to filter undefined results
  scripts: get_abi.pl: don't skip what that ends with wildcards
  scripts: get_abi.pl: Ignore fs/cgroup sysfs nodes earlier
  scripts: get_abi.pl: add a graph to speedup the undefined algorithm

 scripts/get_abi.pl | 327 ++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 320 insertions(+), 7 deletions(-)

-- 
2.31.1

[PATCH v3 2/7] scripts: get_abi.pl: Check for missing symbols at the ABI specs

From: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Date: 2021-09-18 09:52:48

Check for the symbols that exists under /sys but aren't
defined at Documentation/ABI.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 scripts/get_abi.pl | 90 ++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 88 insertions(+), 2 deletions(-)

diff --git a/scripts/get_abi.pl b/scripts/get_abi.pl
index cfc107df59f4..78364c4c4967 100755
--- a/scripts/get_abi.pl
+++ b/scripts/get_abi.pl

@@ -13,7 +13,9 @@ my $help = 0;
 my $man = 0;
 my $debug = 0;
 my $enable_lineno = 0;
+my $show_warnings = 1;
 my $prefix="Documentation/ABI";
+my $sysfs_prefix="/sys";
 
 #
 # If true, assumes that the description is formatted with ReST

@@ -36,7 +38,7 @@ pod2usage(2) if (scalar @ARGV < 1 || @ARGV > 2);
 
 my ($cmd, $arg) = @ARGV;
 
-pod2usage(2) if ($cmd ne "search" && $cmd ne "rest" && $cmd ne "validate");
+pod2usage(2) if ($cmd ne "search" && $cmd ne "rest" && $cmd ne "validate" && $cmd ne "undefined");
 pod2usage(2) if ($cmd eq "search" && !$arg);
 
 require Data::Dumper if ($debug);

@@ -50,6 +52,8 @@ my %symbols;
 sub parse_error($$$$) {
 	my ($file, $ln, $msg, $data) = @_;
 
+	return if (!$show_warnings);
+
 	$data =~ s/\s+$/\n/;
 
 	print STDERR "Warning: file $file#$ln:\n\t$msg";

@@ -521,11 +525,88 @@ sub search_symbols {
 	}
 }
 
+# Exclude /sys/kernel/debug and /sys/kernel/tracing from the search path
+sub skip_debugfs {
+	if (($File::Find::dir =~ m,^/sys/kernel,)) {
+		return grep {!/(debug|tracing)/ } @_;
+	}
+
+	if (($File::Find::dir =~ m,^/sys/fs,)) {
+		return grep {!/(pstore|bpf|fuse)/ } @_;
+	}
+
+	return @_
+}
+
+my %leaf;
+
+my $escape_symbols = qr { ([\x01-\x08\x0e-\x1f\x21-\x29\x2b-\x2d\x3a-\x40\x7b-\xff]) }x;
+sub parse_existing_sysfs {
+	my $file = $File::Find::name;
+
+	my $mode = (stat($file))[2];
+	return if ($mode & S_IFDIR);
+
+	my $leave = $file;
+	$leave =~ s,.*/,,;
+
+	if (defined($leaf{$leave})) {
+		# FIXME: need to check if the path makes sense
+		my $what = $leaf{$leave};
+
+		$what =~ s/,/ /g;
+
+		$what =~ s/\<[^\>]+\>/.*/g;
+		$what =~ s/\{[^\}]+\}/.*/g;
+		$what =~ s/\[[^\]]+\]/.*/g;
+		$what =~ s,/\.\.\./,/.*/,g;
+		$what =~ s,/\*/,/.*/,g;
+
+		$what =~ s/\s+/ /g;
+
+		# Escape all other symbols
+		$what =~ s/$escape_symbols/\\$1/g;
+
+		foreach my $i (split / /,$what) {
+			if ($file =~ m#^$i$#) {
+#				print "$file: $i: OK!\n";
+				return;
+			}
+		}
+
+		print "$file: $leave is defined at $what\n";
+
+		return;
+	}
+
+	print "$file not found.\n";
+}
+
+sub undefined_symbols {
+	foreach my $w (sort keys %data) {
+		foreach my $what (split /\xac /,$w) {
+			my $leave = $what;
+			$leave =~ s,.*/,,;
+
+			if (defined($leaf{$leave})) {
+				$leaf{$leave} .= " " . $what;
+			} else {
+				$leaf{$leave} = $what;
+			}
+		}
+	}
+
+	find({wanted =>\&parse_existing_sysfs, preprocess =>\&skip_debugfs, no_chdir => 1}, $sysfs_prefix);
+}
+
 # Ensure that the prefix will always end with a slash
 # While this is not needed for find, it makes the patch nicer
 # with --enable-lineno
 $prefix =~ s,/?$,/,;
 
+if ($cmd eq "undefined" || $cmd eq "search") {
+	$show_warnings = 0;
+}
 #
 # Parses all ABI files located at $prefix dir
 #

@@ -536,7 +617,9 @@ print STDERR Data::Dumper->Dump([\%data], [qw(*data)]) if ($debug);
 #
 # Handles the command
 #
-if ($cmd eq "search") {
+if ($cmd eq "undefined") {
+	undefined_symbols;
+} elsif ($cmd eq "search") {
 	search_symbols;
 } else {
 	if ($cmd eq "rest") {

@@ -575,6 +658,9 @@ B<rest>                  - output the ABI in ReST markup language
 
 B<validate>              - validate the ABI contents
 
+B<undefined>             - existing symbols at the system that aren't
+                           defined at Documentation/ABI
+
 =back
 
 =head1 OPTIONS

-- 
2.31.1

Re: [PATCH v3 0/7] get_abi.pl: Check for missing symbols at the ABI specs

From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Date: 2021-09-21 16:52:47

On Sat, Sep 18, 2021 at 11:52:10AM +0200, Mauro Carvalho Chehab wrote:

Hi Greg,

Add a new feature at get_abi.pl to optionally check for existing symbols
under /sys that won't match a "What:" inside Documentation/ABI.

Such feature is very useful to detect missing documentation for ABI.

This series brings a major speedup, plus it fixes a few border cases when
matching regexes that end with a ".*" or \d+.

patch 1 changes get_abi.pl logic to handle multiple What: lines, in
order to make the script more robust;

patch 2 adds the basic logic. It runs really quicky (up to 2
seconds), but it doesn't use sysfs softlinks.

Patch 3 adds support for parsing softlinks. It makes the script a
lot slower, making it take a couple of minutes to process the entire
sysfs files. It could be optimized in the future by using a graph,
but, for now, let's keep it simple.

Patch 4 adds an optional parameter to allow filtering the results
using a regex given by the user. When this parameter is used
(which should be the normal usecase), it will only try to find softlinks
if the sysfs node matches a regex.

Patch 5 improves the report by avoiding it to ignore What: that
ends with a wildcard.

Patch 6 is a minor speedup.  On a Dell Precision 5820, after patch 6, 
results are:

	$ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols

	real	2m35.563s
	user	2m34.346s
	sys	0m1.220s
	7595 undefined
	896 undefined_symbols

Patch 7 makes a *huge* speedup: it basically switches a linear O(n^3)
search for links by a logic which handle symlinks using BFS. It
also addresses a border case that was making 'msi-irqs/\d+' regex to
be misparsed. 

After patch 7, it is 11 times faster:

	$ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols

	real	0m14.137s
	user	0m12.795s
	sys	0m1.348s
	7030 undefined
	794 undefined_symbols

(the difference on the number of undefined symbols are due to the fix for
it to properly handle 'msi-irqs/\d+' regex)

-

While this series is independent from Documentation/ABI changes, it
works best when applied from this tree, which also contain ABI fixes
and a couple of additions of frequent missed symbols on my machine:

    https://git.kernel.org/pub/scm/linux/kernel/git/mchehab/devel.git/log/?h=get_undefined_abi_v3

I've taken all of these, but get_abi.pl seems to be stuck in an endless
loop or something.  I gave up and stopped it after 14 minutes.  It had
stopped printing out anything after finding all of the pci attributes
that are not documented :)

Anything I can do to help debug this?

thanks,

greg k-h

Re: [PATCH v3 0/7] get_abi.pl: Check for missing symbols at the ABI specs

From: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Date: 2021-09-21 18:16:43

Em Tue, 21 Sep 2021 18:52:42 +0200
Greg Kroah-Hartman [off-list ref] escreveu:

On Sat, Sep 18, 2021 at 11:52:10AM +0200, Mauro Carvalho Chehab wrote:

quoted

Hi Greg,

Add a new feature at get_abi.pl to optionally check for existing symbols
under /sys that won't match a "What:" inside Documentation/ABI.

Such feature is very useful to detect missing documentation for ABI.

This series brings a major speedup, plus it fixes a few border cases when
matching regexes that end with a ".*" or \d+.

patch 1 changes get_abi.pl logic to handle multiple What: lines, in
order to make the script more robust;

patch 2 adds the basic logic. It runs really quicky (up to 2
seconds), but it doesn't use sysfs softlinks.

Patch 3 adds support for parsing softlinks. It makes the script a
lot slower, making it take a couple of minutes to process the entire
sysfs files. It could be optimized in the future by using a graph,
but, for now, let's keep it simple.

Patch 4 adds an optional parameter to allow filtering the results
using a regex given by the user. When this parameter is used
(which should be the normal usecase), it will only try to find softlinks
if the sysfs node matches a regex.

Patch 5 improves the report by avoiding it to ignore What: that
ends with a wildcard.

Patch 6 is a minor speedup.  On a Dell Precision 5820, after patch 6, 
results are:

	$ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols

	real	2m35.563s
	user	2m34.346s
	sys	0m1.220s
	7595 undefined
	896 undefined_symbols

Patch 7 makes a *huge* speedup: it basically switches a linear O(n^3)
search for links by a logic which handle symlinks using BFS. It
also addresses a border case that was making 'msi-irqs/\d+' regex to
be misparsed. 

After patch 7, it is 11 times faster:

	$ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols

	real	0m14.137s
	user	0m12.795s
	sys	0m1.348s
	7030 undefined
	794 undefined_symbols

(the difference on the number of undefined symbols are due to the fix for
it to properly handle 'msi-irqs/\d+' regex)

-

While this series is independent from Documentation/ABI changes, it
works best when applied from this tree, which also contain ABI fixes
and a couple of additions of frequent missed symbols on my machine:

    https://git.kernel.org/pub/scm/linux/kernel/git/mchehab/devel.git/log/?h=get_undefined_abi_v3

I've taken all of these, but get_abi.pl seems to be stuck in an endless
loop or something.  I gave up and stopped it after 14 minutes.  It had
stopped printing out anything after finding all of the pci attributes
that are not documented :)

It is probably not an endless loop, just there are too many vars to
check on your system, which could make it really slow.

The way the search algorithm works is that reduces the number of regex 
expressions that will be checked for a given file entry at sysfs. It 
does that by looking at the devnode name. For instance, when it checks for
this file:

	/sys/bus/pci/drivers/iosf_mbi_pci/bind

The logic will seek only the "What:" expressions that end with "bind".
Currently, there are just two What expressions for it[1]:

	What: /sys/bus/fsl\-mc/drivers/.*/bind
	What: /sys/bus/pci/drivers/.*/bind

It will then run an O(n²) algorithm to seek:

		foreach my $a (@names) {
                       foreach my $w (split /\xac/, $what) {
                               if ($a =~ m#^$w$#) {
					exact = 1;
                                        last;
                                }
			}
		}

Which runs quickly, when there are few regexs to seek. There are, 
however, some What: expressions that end with a wildcard. Those are
harder to process. Right now, they're all grouped together, which
makes them slower. Most of the processing time are spent on those.

I'm working right now on some strategy to also speed up the search 
for them. Once I get something better, I'll send a patch series.

--

[1] On a side note, there are currently some problems with the What:
    definitions for bind/unbind, as:

	- it doesn't match all PCI devices;
	- it doesn't match ACPI and other buses that also export
	  bind/unbind.

Anything I can do to help debug this?

There are two parameters that can help to identify the issue:

a) You can add a "--show-hints" parameter. This turns on some 
   prints that may help to identify what the script is doing.
   It is not really a debug option, but it helps to identify
   when some regexes are failing.

b) You can limit the What expressions that will be parsed with:
	   --search-string <something>

You can combine both. For instance, if you want to make it
a lot more verbose, you could run it as:

	./scripts/get_abi.pl undefined --search-string /sys --show-hints

The script will then print all regexes that will be checked, and when
actually checking for the missing vars, it will print all names for
a given entry at sysfs.

So, if you want to know how an i2c bind has been validated, you
could do:

	$ ./scripts/get_abi.pl undefined --search-string i2c/.*/bind --show-hints
	--> /sys/bus/i2c/drivers/dummy/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-3/i2c-14/subsystem/drivers/dummy/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-4/i2c-15/subsystem/drivers/dummy/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0036/subsystem/drivers/dummy/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0037/driver/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-2/i2c-13/subsystem/drivers/dummy/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0050/subsystem/drivers/dummy/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-10/subsystem/drivers/dummy/bind
	--> /sys/devices/pci0000:00/0000:00:02.0/i2c-5/subsystem/drivers/dummy/bind
	--> /sys/devices/pci0000:00/0000:00:02.0/i2c-3/subsystem/drivers/dummy/bind
	--> /sys/devices/pci0000:00/0000:00:15.1/i2c_designware.1/i2c-1/subsystem/drivers/dummy/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-1/i2c-12/subsystem/drivers/dummy/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0037/subsystem/drivers/dummy/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-8/subsystem/drivers/dummy/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-9/subsystem/drivers/dummy/bind
	--> /sys/devices/pci0000:00/0000:00:15.2/i2c_designware.2/i2c-2/subsystem/drivers/dummy/bind
	--> /sys/devices/pci0000:00/0000:00:15.0/i2c_designware.0/i2c-0/subsystem/drivers/dummy/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0036/driver/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/subsystem/drivers/dummy/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-7/subsystem/drivers/dummy/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-6/subsystem/drivers/dummy/bind
	--> /sys/devices/pci0000:00/0000:00:02.0/i2c-4/subsystem/drivers/dummy/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-11/subsystem/drivers/dummy/bind
	    more likely regexes:
		/sys/bus/fsl\-mc/drivers/.*/bind
		/sys/bus/pci/drivers/.*/bind
	--> /sys/bus/i2c/drivers/axp20x-i2c/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-3/i2c-14/subsystem/drivers/axp20x-i2c/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-4/i2c-15/subsystem/drivers/axp20x-i2c/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0036/subsystem/drivers/axp20x-i2c/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-2/i2c-13/subsystem/drivers/axp20x-i2c/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0050/subsystem/drivers/axp20x-i2c/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-10/subsystem/drivers/axp20x-i2c/bind
	--> /sys/devices/pci0000:00/0000:00:02.0/i2c-5/subsystem/drivers/axp20x-i2c/bind
	--> /sys/devices/pci0000:00/0000:00:02.0/i2c-3/subsystem/drivers/axp20x-i2c/bind
	--> /sys/devices/pci0000:00/0000:00:15.1/i2c_designware.1/i2c-1/subsystem/drivers/axp20x-i2c/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-1/i2c-12/subsystem/drivers/axp20x-i2c/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0037/subsystem/drivers/axp20x-i2c/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-8/subsystem/drivers/axp20x-i2c/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-9/subsystem/drivers/axp20x-i2c/bind
	--> /sys/devices/pci0000:00/0000:00:15.2/i2c_designware.2/i2c-2/subsystem/drivers/axp20x-i2c/bind
	--> /sys/devices/pci0000:00/0000:00:15.0/i2c_designware.0/i2c-0/subsystem/drivers/axp20x-i2c/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/subsystem/drivers/axp20x-i2c/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-7/subsystem/drivers/axp20x-i2c/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-6/subsystem/drivers/axp20x-i2c/bind
	--> /sys/devices/pci0000:00/0000:00:02.0/i2c-4/subsystem/drivers/axp20x-i2c/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-11/subsystem/drivers/axp20x-i2c/bind
	    more likely regexes:
		/sys/bus/fsl\-mc/drivers/.*/bind
		/sys/bus/pci/drivers/.*/bind
	--> /sys/bus/i2c/drivers/smbus_alert/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-3/i2c-14/subsystem/drivers/smbus_alert/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-4/i2c-15/subsystem/drivers/smbus_alert/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0036/subsystem/drivers/smbus_alert/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-2/i2c-13/subsystem/drivers/smbus_alert/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0050/subsystem/drivers/smbus_alert/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-10/subsystem/drivers/smbus_alert/bind
	--> /sys/devices/pci0000:00/0000:00:02.0/i2c-5/subsystem/drivers/smbus_alert/bind
	--> /sys/devices/pci0000:00/0000:00:02.0/i2c-3/subsystem/drivers/smbus_alert/bind
	--> /sys/devices/pci0000:00/0000:00:15.1/i2c_designware.1/i2c-1/subsystem/drivers/smbus_alert/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-1/i2c-12/subsystem/drivers/smbus_alert/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0037/subsystem/drivers/smbus_alert/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-8/subsystem/drivers/smbus_alert/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-9/subsystem/drivers/smbus_alert/bind
	--> /sys/devices/pci0000:00/0000:00:15.2/i2c_designware.2/i2c-2/subsystem/drivers/smbus_alert/bind
	--> /sys/devices/pci0000:00/0000:00:15.0/i2c_designware.0/i2c-0/subsystem/drivers/smbus_alert/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/subsystem/drivers/smbus_alert/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-7/subsystem/drivers/smbus_alert/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-6/subsystem/drivers/smbus_alert/bind
	--> /sys/devices/pci0000:00/0000:00:02.0/i2c-4/subsystem/drivers/smbus_alert/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-11/subsystem/drivers/smbus_alert/bind
	--> /sys/module/i2c_smbus/drivers/i2c:smbus_alert/bind
	    more likely regexes:
		/sys/bus/fsl\-mc/drivers/.*/bind
		/sys/bus/pci/drivers/.*/bind
	--> /sys/bus/i2c/drivers/ee1004/bind
	--> /sys/module/ee1004/drivers/i2c:ee1004/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-3/i2c-14/subsystem/drivers/ee1004/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-4/i2c-15/subsystem/drivers/ee1004/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0036/subsystem/drivers/ee1004/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-2/i2c-13/subsystem/drivers/ee1004/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0050/subsystem/drivers/ee1004/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-10/subsystem/drivers/ee1004/bind
	--> /sys/devices/pci0000:00/0000:00:02.0/i2c-5/subsystem/drivers/ee1004/bind
	--> /sys/devices/pci0000:00/0000:00:02.0/i2c-3/subsystem/drivers/ee1004/bind
	--> /sys/devices/pci0000:00/0000:00:15.1/i2c_designware.1/i2c-1/subsystem/drivers/ee1004/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-1/i2c-12/subsystem/drivers/ee1004/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0037/subsystem/drivers/ee1004/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-8/subsystem/drivers/ee1004/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-9/subsystem/drivers/ee1004/bind
	--> /sys/devices/pci0000:00/0000:00:15.2/i2c_designware.2/i2c-2/subsystem/drivers/ee1004/bind
	--> /sys/devices/pci0000:00/0000:00:15.0/i2c_designware.0/i2c-0/subsystem/drivers/ee1004/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/subsystem/drivers/ee1004/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-7/subsystem/drivers/ee1004/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-6/subsystem/drivers/ee1004/bind
	--> /sys/devices/pci0000:00/0000:00:02.0/i2c-4/subsystem/drivers/ee1004/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-11/subsystem/drivers/ee1004/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0050/driver/bind
	    more likely regexes:
		/sys/bus/fsl\-mc/drivers/.*/bind
		/sys/bus/pci/drivers/.*/bind
	--> /sys/bus/i2c/drivers/intel_soc_pmic_i2c/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-3/i2c-14/subsystem/drivers/intel_soc_pmic_i2c/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-4/i2c-15/subsystem/drivers/intel_soc_pmic_i2c/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0036/subsystem/drivers/intel_soc_pmic_i2c/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-2/i2c-13/subsystem/drivers/intel_soc_pmic_i2c/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0050/subsystem/drivers/intel_soc_pmic_i2c/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-10/subsystem/drivers/intel_soc_pmic_i2c/bind
	--> /sys/devices/pci0000:00/0000:00:02.0/i2c-5/subsystem/drivers/intel_soc_pmic_i2c/bind
	--> /sys/devices/pci0000:00/0000:00:02.0/i2c-3/subsystem/drivers/intel_soc_pmic_i2c/bind
	--> /sys/devices/pci0000:00/0000:00:15.1/i2c_designware.1/i2c-1/subsystem/drivers/intel_soc_pmic_i2c/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-1/i2c-12/subsystem/drivers/intel_soc_pmic_i2c/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0037/subsystem/drivers/intel_soc_pmic_i2c/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-8/subsystem/drivers/intel_soc_pmic_i2c/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-9/subsystem/drivers/intel_soc_pmic_i2c/bind
	--> /sys/devices/pci0000:00/0000:00:15.2/i2c_designware.2/i2c-2/subsystem/drivers/intel_soc_pmic_i2c/bind
	--> /sys/devices/pci0000:00/0000:00:15.0/i2c_designware.0/i2c-0/subsystem/drivers/intel_soc_pmic_i2c/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/subsystem/drivers/intel_soc_pmic_i2c/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-7/subsystem/drivers/intel_soc_pmic_i2c/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-6/subsystem/drivers/intel_soc_pmic_i2c/bind
	--> /sys/devices/pci0000:00/0000:00:02.0/i2c-4/subsystem/drivers/intel_soc_pmic_i2c/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-11/subsystem/drivers/intel_soc_pmic_i2c/bind
	    more likely regexes:
		/sys/bus/fsl\-mc/drivers/.*/bind
		/sys/bus/pci/drivers/.*/bind
	--> /sys/bus/i2c/drivers/tps68470/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-3/i2c-14/subsystem/drivers/tps68470/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-4/i2c-15/subsystem/drivers/tps68470/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0036/subsystem/drivers/tps68470/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-2/i2c-13/subsystem/drivers/tps68470/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0050/subsystem/drivers/tps68470/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-10/subsystem/drivers/tps68470/bind
	--> /sys/devices/pci0000:00/0000:00:02.0/i2c-5/subsystem/drivers/tps68470/bind
	--> /sys/devices/pci0000:00/0000:00:02.0/i2c-3/subsystem/drivers/tps68470/bind
	--> /sys/devices/pci0000:00/0000:00:15.1/i2c_designware.1/i2c-1/subsystem/drivers/tps68470/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-1/i2c-12/subsystem/drivers/tps68470/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0037/subsystem/drivers/tps68470/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-8/subsystem/drivers/tps68470/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-9/subsystem/drivers/tps68470/bind
	--> /sys/devices/pci0000:00/0000:00:15.2/i2c_designware.2/i2c-2/subsystem/drivers/tps68470/bind
	--> /sys/devices/pci0000:00/0000:00:15.0/i2c_designware.0/i2c-0/subsystem/drivers/tps68470/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/subsystem/drivers/tps68470/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-7/subsystem/drivers/tps68470/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-6/subsystem/drivers/tps68470/bind
	--> /sys/devices/pci0000:00/0000:00:02.0/i2c-4/subsystem/drivers/tps68470/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-11/subsystem/drivers/tps68470/bind
	    more likely regexes:
		/sys/bus/fsl\-mc/drivers/.*/bind
		/sys/bus/pci/drivers/.*/bind
	--> /sys/bus/i2c/drivers/CHT Whiskey Cove PMIC/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-3/i2c-14/subsystem/drivers/CHT Whiskey Cove PMIC/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-4/i2c-15/subsystem/drivers/CHT Whiskey Cove PMIC/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0036/subsystem/drivers/CHT Whiskey Cove PMIC/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-2/i2c-13/subsystem/drivers/CHT Whiskey Cove PMIC/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0050/subsystem/drivers/CHT Whiskey Cove PMIC/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-10/subsystem/drivers/CHT Whiskey Cove PMIC/bind
	--> /sys/devices/pci0000:00/0000:00:02.0/i2c-5/subsystem/drivers/CHT Whiskey Cove PMIC/bind
	--> /sys/devices/pci0000:00/0000:00:02.0/i2c-3/subsystem/drivers/CHT Whiskey Cove PMIC/bind
	--> /sys/devices/pci0000:00/0000:00:15.1/i2c_designware.1/i2c-1/subsystem/drivers/CHT Whiskey Cove PMIC/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-1/i2c-12/subsystem/drivers/CHT Whiskey Cove PMIC/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0037/subsystem/drivers/CHT Whiskey Cove PMIC/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-8/subsystem/drivers/CHT Whiskey Cove PMIC/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-9/subsystem/drivers/CHT Whiskey Cove PMIC/bind
	--> /sys/devices/pci0000:00/0000:00:15.2/i2c_designware.2/i2c-2/subsystem/drivers/CHT Whiskey Cove PMIC/bind
	--> /sys/devices/pci0000:00/0000:00:15.0/i2c_designware.0/i2c-0/subsystem/drivers/CHT Whiskey Cove PMIC/bind
	--> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/subsystem/drivers/CHT Whiskey Cove PMIC/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-7/subsystem/drivers/CHT Whiskey Cove PMIC/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-6/subsystem/drivers/CHT Whiskey Cove PMIC/bind
	--> /sys/devices/pci0000:00/0000:00:02.0/i2c-4/subsystem/drivers/CHT Whiskey Cove PMIC/bind
	--> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-11/subsystem/drivers/CHT Whiskey Cove PMIC/bind
	    more likely regexes:
		/sys/bus/fsl\-mc/drivers/.*/bind
		/sys/bus/pci/drivers/.*/bind

Btw, on the above example, I have already a patch addressing it
(see enclosed). I intend to submit it on a newer patch series.

Thanks,
Mauro

[PATCH] ABI: sysfs-bus-pci: add a alternative What fields

There are some PCI ABI that aren't shown under:

	/sys/bus/pci/drivers/.../

Because they're registered with a different class. That's
the case of, for instance:

	/sys/bus/i2c/drivers/CHT Whiskey Cove PMIC/unbind

This one is not present under /sys/bus/pci:

	$ find /sys/bus/pci -name 'CHT Whiskey Cove PMIC'

Although clearly this is provided by a PCI driver:

	/sys/devices/pci0000:00/0000:00:02.0/i2c-4/subsystem/drivers/CHT Whiskey Cove PMIC/unbind

So, add an altertate What location in order to match bind/unbind
to such devices.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

diff --git a/Documentation/ABI/testing/sysfs-bus-pci b/Documentation/ABI/testing/sysfs-bus-pci
index 1da4c8db3a9e..f4efbcb0b18c 100644
--- a/Documentation/ABI/testing/sysfs-bus-pci
+++ b/Documentation/ABI/testing/sysfs-bus-pci

@@ -1,4 +1,5 @@
 What:		/sys/bus/pci/drivers/.../bind
+What:		/sys/devices/pciX/.../bind
 Date:		December 2003
 Contact:	linux-pci@vger.kernel.org
 Description:

@@ -14,6 +15,7 @@ Description:
 		(Note: kernels before 2.6.28 may require echo -n).
 
 What:		/sys/bus/pci/drivers/.../unbind
+What:		/sys/devices/pciX/.../unbind
 Date:		December 2003
 Contact:	linux-pci@vger.kernel.org
 Description:

@@ -29,6 +31,7 @@ Description:
 		(Note: kernels before 2.6.28 may require echo -n).
 
 What:		/sys/bus/pci/drivers/.../new_id
+What:		/sys/devices/pciX/.../new_id
 Date:		December 2003
 Contact:	linux-pci@vger.kernel.org
 Description:

@@ -47,6 +50,7 @@ Description:
 		  # echo "8086 10f5" > /sys/bus/pci/drivers/foo/new_id
 
 What:		/sys/bus/pci/drivers/.../remove_id
+What:		/sys/devices/pciX/.../remove_id
 Date:		February 2009
 Contact:	Chris Wright <chrisw@sous-sol.org>
 Description:

Re: [PATCH v3 0/7] get_abi.pl: Check for missing symbols at the ABI specs

From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Date: 2021-09-22 05:43:49

On Tue, Sep 21, 2021 at 08:16:33PM +0200, Mauro Carvalho Chehab wrote:

Em Tue, 21 Sep 2021 18:52:42 +0200
Greg Kroah-Hartman [off-list ref] escreveu:

quoted

On Sat, Sep 18, 2021 at 11:52:10AM +0200, Mauro Carvalho Chehab wrote:

quoted

Hi Greg,

Add a new feature at get_abi.pl to optionally check for existing symbols
under /sys that won't match a "What:" inside Documentation/ABI.

Such feature is very useful to detect missing documentation for ABI.

This series brings a major speedup, plus it fixes a few border cases when
matching regexes that end with a ".*" or \d+.

patch 1 changes get_abi.pl logic to handle multiple What: lines, in
order to make the script more robust;

patch 2 adds the basic logic. It runs really quicky (up to 2
seconds), but it doesn't use sysfs softlinks.

Patch 3 adds support for parsing softlinks. It makes the script a
lot slower, making it take a couple of minutes to process the entire
sysfs files. It could be optimized in the future by using a graph,
but, for now, let's keep it simple.

Patch 4 adds an optional parameter to allow filtering the results
using a regex given by the user. When this parameter is used
(which should be the normal usecase), it will only try to find softlinks
if the sysfs node matches a regex.

Patch 5 improves the report by avoiding it to ignore What: that
ends with a wildcard.

Patch 6 is a minor speedup.  On a Dell Precision 5820, after patch 6, 
results are:

	$ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols

	real	2m35.563s
	user	2m34.346s
	sys	0m1.220s
	7595 undefined
	896 undefined_symbols

Patch 7 makes a *huge* speedup: it basically switches a linear O(n^3)
search for links by a logic which handle symlinks using BFS. It
also addresses a border case that was making 'msi-irqs/\d+' regex to
be misparsed. 

After patch 7, it is 11 times faster:

	$ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols

	real	0m14.137s
	user	0m12.795s
	sys	0m1.348s
	7030 undefined
	794 undefined_symbols

(the difference on the number of undefined symbols are due to the fix for
it to properly handle 'msi-irqs/\d+' regex)

-

While this series is independent from Documentation/ABI changes, it
works best when applied from this tree, which also contain ABI fixes
and a couple of additions of frequent missed symbols on my machine:

    https://git.kernel.org/pub/scm/linux/kernel/git/mchehab/devel.git/log/?h=get_undefined_abi_v3

I've taken all of these, but get_abi.pl seems to be stuck in an endless
loop or something.  I gave up and stopped it after 14 minutes.  It had
stopped printing out anything after finding all of the pci attributes
that are not documented :)

It is probably not an endless loop, just there are too many vars to
check on your system, which could make it really slow.

Ah, yes, I ran it overnight and got the following:

$ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols

real	29m39.503s
user	29m37.556s
sys	0m0.851s
26669 undefined
765 undefined_symbols

The way the search algorithm works is that reduces the number of regex 
expressions that will be checked for a given file entry at sysfs. It 
does that by looking at the devnode name. For instance, when it checks for
this file:

	/sys/bus/pci/drivers/iosf_mbi_pci/bind

The logic will seek only the "What:" expressions that end with "bind".
Currently, there are just two What expressions for it[1]:

	What: /sys/bus/fsl\-mc/drivers/.*/bind
	What: /sys/bus/pci/drivers/.*/bind

It will then run an O(n²) algorithm to seek:

		foreach my $a (@names) {
                       foreach my $w (split /\xac/, $what) {
                               if ($a =~ m#^$w$#) {
					exact = 1;
                                        last;
                                }
			}
		}

Which runs quickly, when there are few regexs to seek. There are, 
however, some What: expressions that end with a wildcard. Those are
harder to process. Right now, they're all grouped together, which
makes them slower. Most of the processing time are spent on those.

I'm working right now on some strategy to also speed up the search 
for them. Once I get something better, I'll send a patch series.

--

[1] On a side note, there are currently some problems with the What:
    definitions for bind/unbind, as:

	- it doesn't match all PCI devices;
	- it doesn't match ACPI and other buses that also export
	  bind/unbind.

quoted

Anything I can do to help debug this?

There are two parameters that can help to identify the issue:

a) You can add a "--show-hints" parameter. This turns on some 
   prints that may help to identify what the script is doing.
   It is not really a debug option, but it helps to identify
   when some regexes are failing.

b) You can limit the What expressions that will be parsed with:
	   --search-string <something>

You can combine both. For instance, if you want to make it
a lot more verbose, you could run it as:

	./scripts/get_abi.pl undefined --search-string /sys --show-hints

Let me run this and time stamp it to see where it is getting hung up on.
Give it another 30 minutes :)

thanks,

greg k-hj

`h`	back out one level
`j`	next message in thread
`k`	previous message in thread
`l`	drill in
`Esc`	close help / fold thread tree
`?`	toggle this help