[PATCH 1/4] perf test: Fix non-bash issue with stat bpf counters

Subsystems: performance events subsystem, the rest

STALE1881d LANDED

Landed in mainline as 482698c2f848 on 2021-06-19.

13 messages, 3 authors, 2021-06-21 · open the first message on its own page

[PATCH 1/4] perf test: Fix non-bash issue with stat bpf counters

From: Ian Rogers <irogers@google.com>
Date: 2021-06-17 18:42:31

$(( .. )) is a bash feature but the test's interpreter is !/bin/sh,
switch the code to use expr.

Signed-off-by: Ian Rogers <irogers@google.com>
---
 tools/perf/tests/shell/stat_bpf_counters.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tools/perf/tests/shell/stat_bpf_counters.sh b/tools/perf/tests/shell/stat_bpf_counters.sh
index 22eb31e48ca7..2f9948b3d943 100755
--- a/tools/perf/tests/shell/stat_bpf_counters.sh
+++ b/tools/perf/tests/shell/stat_bpf_counters.sh
@@ -11,9 +11,9 @@ compare_number()
        second_num=$2
 
        # upper bound is first_num * 110%
-       upper=$(( $first_num + $first_num / 10 ))
+       upper=$(expr $first_num + $first_num / 10 )
        # lower bound is first_num * 90%
-       lower=$(( $first_num - $first_num / 10 ))
+       lower=$(expr $first_num - $first_num / 10 )
 
        if [ $second_num -gt $upper ] || [ $second_num -lt $lower ]; then
                echo "The difference between $first_num and $second_num are greater than 10%."
-- 
2.32.0.288.g62a8d224e6-goog

[PATCH 2/4] perf test: Pass the verbose option to shell tests

From: Ian Rogers <irogers@google.com>
Date: 2021-06-17 18:42:39

Having a verbose option will allow shell tests to provide extra failure
details when the fail or skip.

Signed-off-by: Ian Rogers <irogers@google.com>
---
 tools/perf/tests/builtin-test.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index cbbfe48ab802..a8160b1684de 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -577,11 +577,14 @@ struct shell_test {
 static int shell_test__run(struct test *test, int subdir __maybe_unused)
 {
 	int err;
-	char script[PATH_MAX];
+	char script[PATH_MAX + 3];
 	struct shell_test *st = test->priv;
 
 	path__join(script, sizeof(script), st->dir, st->file);
 
+	if (verbose)
+		strncat(script, " -v", sizeof(script));
+
 	err = system(script);
 	if (!err)
 		return TEST_OK;
-- 
2.32.0.288.g62a8d224e6-goog

[PATCH 3/4] perf test: Add verbose skip output for bpf counters

From: Ian Rogers <irogers@google.com>
Date: 2021-06-17 18:42:43

Provide additional context for when the stat bpf counters test skips.

Signed-off-by: Ian Rogers <irogers@google.com>
---
 tools/perf/tests/shell/stat_bpf_counters.sh | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/tools/perf/tests/shell/stat_bpf_counters.sh b/tools/perf/tests/shell/stat_bpf_counters.sh
index 2f9948b3d943..81d61b6e1208 100755
--- a/tools/perf/tests/shell/stat_bpf_counters.sh
+++ b/tools/perf/tests/shell/stat_bpf_counters.sh
@@ -22,7 +22,13 @@ compare_number()
 }
 
 # skip if --bpf-counters is not supported
-perf stat --bpf-counters true > /dev/null 2>&1 || exit 2
+if ! perf stat --bpf-counters true > /dev/null 2>&1; then
+	if [ "$1" == "-v" ]; then
+		echo "Skipping: --bpf-counters not supported"
+		perf --no-pager stat --bpf-counters true || true
+	fi
+	exit 2
+fi
 
 base_cycles=$(perf stat --no-big-num -e cycles -- perf bench sched messaging -g 1 -l 100 -t 2>&1 | awk '/cycles/ {print $1}')
 bpf_cycles=$(perf stat --no-big-num --bpf-counters -e cycles -- perf bench sched messaging -g 1 -l 100 -t 2>&1 | awk '/cycles/ {print $1}')
-- 
2.32.0.288.g62a8d224e6-goog

[PATCH 4/4] perf test: Make stat bpf counters test more robust

From: Ian Rogers <irogers@google.com>
Date: 2021-06-17 18:42:46

If the test is run on a hypervisor then the cycles event may not be
counted, skip the test in this situation. Fail the test if cycles are
not counted in the subsequent bpf counter run.

Signed-off-by: Ian Rogers <irogers@google.com>
---
 tools/perf/tests/shell/stat_bpf_counters.sh | 8 ++++++++
 1 file changed, 8 insertions(+)
diff --git a/tools/perf/tests/shell/stat_bpf_counters.sh b/tools/perf/tests/shell/stat_bpf_counters.sh
index 81d61b6e1208..2aed20dc2262 100755
--- a/tools/perf/tests/shell/stat_bpf_counters.sh
+++ b/tools/perf/tests/shell/stat_bpf_counters.sh
@@ -31,7 +31,15 @@ if ! perf stat --bpf-counters true > /dev/null 2>&1; then
 fi
 
 base_cycles=$(perf stat --no-big-num -e cycles -- perf bench sched messaging -g 1 -l 100 -t 2>&1 | awk '/cycles/ {print $1}')
+if [ "$base_cycles" == "<not" ]; then
+	echo "Skipping: cycles event not counted"
+	exit 2
+fi
 bpf_cycles=$(perf stat --no-big-num --bpf-counters -e cycles -- perf bench sched messaging -g 1 -l 100 -t 2>&1 | awk '/cycles/ {print $1}')
+if [ "$bpf_cycles" == "<not" ]; then
+	echo "Failed: cycles not counted with --bpf-counters"
+	exit 1
+fi
 
 compare_number $base_cycles $bpf_cycles
 exit 0
-- 
2.32.0.288.g62a8d224e6-goog

Re: [PATCH 1/4] perf test: Fix non-bash issue with stat bpf counters

From: Arnaldo Carvalho de Melo <acme@kernel.org>
Date: 2021-06-17 19:18:38

Em Thu, Jun 17, 2021 at 11:42:13AM -0700, Ian Rogers escreveu:
$(( .. )) is a bash feature but the test's interpreter is !/bin/sh,
switch the code to use expr.

Thanks, applied to perf/urgent.

- Arnaldo

 
quoted hunk
Signed-off-by: Ian Rogers <irogers@google.com>
---
 tools/perf/tests/shell/stat_bpf_counters.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tools/perf/tests/shell/stat_bpf_counters.sh b/tools/perf/tests/shell/stat_bpf_counters.sh
index 22eb31e48ca7..2f9948b3d943 100755
--- a/tools/perf/tests/shell/stat_bpf_counters.sh
+++ b/tools/perf/tests/shell/stat_bpf_counters.sh
@@ -11,9 +11,9 @@ compare_number()
        second_num=$2
 
        # upper bound is first_num * 110%
-       upper=$(( $first_num + $first_num / 10 ))
+       upper=$(expr $first_num + $first_num / 10 )
        # lower bound is first_num * 90%
-       lower=$(( $first_num - $first_num / 10 ))
+       lower=$(expr $first_num - $first_num / 10 )
 
        if [ $second_num -gt $upper ] || [ $second_num -lt $lower ]; then
                echo "The difference between $first_num and $second_num are greater than 10%."
-- 
2.32.0.288.g62a8d224e6-goog
-- 

- Arnaldo

Re: [PATCH 2/4] perf test: Pass the verbose option to shell tests

From: Arnaldo Carvalho de Melo <acme@kernel.org>
Date: 2021-06-17 19:19:12

Em Thu, Jun 17, 2021 at 11:42:14AM -0700, Ian Rogers escreveu:
Having a verbose option will allow shell tests to provide extra failure
details when the fail or skip.
 

Thanks, applied to perf/core.

- Arnaldo
quoted hunk
Signed-off-by: Ian Rogers <irogers@google.com>
---
 tools/perf/tests/builtin-test.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index cbbfe48ab802..a8160b1684de 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -577,11 +577,14 @@ struct shell_test {
 static int shell_test__run(struct test *test, int subdir __maybe_unused)
 {
 	int err;
-	char script[PATH_MAX];
+	char script[PATH_MAX + 3];
 	struct shell_test *st = test->priv;
 
 	path__join(script, sizeof(script), st->dir, st->file);
 
+	if (verbose)
+		strncat(script, " -v", sizeof(script));
+
 	err = system(script);
 	if (!err)
 		return TEST_OK;
-- 
2.32.0.288.g62a8d224e6-goog
-- 

- Arnaldo

Re: [PATCH 3/4] perf test: Add verbose skip output for bpf counters

From: Arnaldo Carvalho de Melo <acme@kernel.org>
Date: 2021-06-17 19:20:42

Em Thu, Jun 17, 2021 at 11:42:15AM -0700, Ian Rogers escreveu:
Provide additional context for when the stat bpf counters test skips.
Thanks, applied.

- Arnaldo

 
quoted hunk
Signed-off-by: Ian Rogers <irogers@google.com>
---
 tools/perf/tests/shell/stat_bpf_counters.sh | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/tools/perf/tests/shell/stat_bpf_counters.sh b/tools/perf/tests/shell/stat_bpf_counters.sh
index 2f9948b3d943..81d61b6e1208 100755
--- a/tools/perf/tests/shell/stat_bpf_counters.sh
+++ b/tools/perf/tests/shell/stat_bpf_counters.sh
@@ -22,7 +22,13 @@ compare_number()
 }
 
 # skip if --bpf-counters is not supported
-perf stat --bpf-counters true > /dev/null 2>&1 || exit 2
+if ! perf stat --bpf-counters true > /dev/null 2>&1; then
+	if [ "$1" == "-v" ]; then
+		echo "Skipping: --bpf-counters not supported"
+		perf --no-pager stat --bpf-counters true || true
+	fi
+	exit 2
+fi
 
 base_cycles=$(perf stat --no-big-num -e cycles -- perf bench sched messaging -g 1 -l 100 -t 2>&1 | awk '/cycles/ {print $1}')
 bpf_cycles=$(perf stat --no-big-num --bpf-counters -e cycles -- perf bench sched messaging -g 1 -l 100 -t 2>&1 | awk '/cycles/ {print $1}')
-- 
2.32.0.288.g62a8d224e6-goog
-- 

- Arnaldo

Re: [PATCH 4/4] perf test: Make stat bpf counters test more robust

From: Arnaldo Carvalho de Melo <acme@kernel.org>
Date: 2021-06-17 19:21:25

Em Thu, Jun 17, 2021 at 11:42:16AM -0700, Ian Rogers escreveu:
If the test is run on a hypervisor then the cycles event may not be
counted, skip the test in this situation. Fail the test if cycles are
not counted in the subsequent bpf counter run.
Thanks, applied.

- Arnaldo

 
quoted hunk
Signed-off-by: Ian Rogers <irogers@google.com>
---
 tools/perf/tests/shell/stat_bpf_counters.sh | 8 ++++++++
 1 file changed, 8 insertions(+)
diff --git a/tools/perf/tests/shell/stat_bpf_counters.sh b/tools/perf/tests/shell/stat_bpf_counters.sh
index 81d61b6e1208..2aed20dc2262 100755
--- a/tools/perf/tests/shell/stat_bpf_counters.sh
+++ b/tools/perf/tests/shell/stat_bpf_counters.sh
@@ -31,7 +31,15 @@ if ! perf stat --bpf-counters true > /dev/null 2>&1; then
 fi
 
 base_cycles=$(perf stat --no-big-num -e cycles -- perf bench sched messaging -g 1 -l 100 -t 2>&1 | awk '/cycles/ {print $1}')
+if [ "$base_cycles" == "<not" ]; then
+	echo "Skipping: cycles event not counted"
+	exit 2
+fi
 bpf_cycles=$(perf stat --no-big-num --bpf-counters -e cycles -- perf bench sched messaging -g 1 -l 100 -t 2>&1 | awk '/cycles/ {print $1}')
+if [ "$bpf_cycles" == "<not" ]; then
+	echo "Failed: cycles not counted with --bpf-counters"
+	exit 1
+fi
 
 compare_number $base_cycles $bpf_cycles
 exit 0
-- 
2.32.0.288.g62a8d224e6-goog
-- 

- Arnaldo

Re: [PATCH 2/4] perf test: Pass the verbose option to shell tests

From: Arnaldo Carvalho de Melo <acme@kernel.org>
Date: 2021-06-18 16:49:48

Em Thu, Jun 17, 2021 at 04:19:06PM -0300, Arnaldo Carvalho de Melo escreveu:
Em Thu, Jun 17, 2021 at 11:42:14AM -0700, Ian Rogers escreveu:
quoted
Having a verbose option will allow shell tests to provide extra failure
details when the fail or skip.
 

Thanks, applied to perf/core.

- Arnaldo
quoted
Signed-off-by: Ian Rogers <irogers@google.com>
---
 tools/perf/tests/builtin-test.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index cbbfe48ab802..a8160b1684de 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -577,11 +577,14 @@ struct shell_test {
 static int shell_test__run(struct test *test, int subdir __maybe_unused)
 {
 	int err;
-	char script[PATH_MAX];
+	char script[PATH_MAX + 3];
 	struct shell_test *st = test->priv;
 
 	path__join(script, sizeof(script), st->dir, st->file);
probably you need to add a  '- 3' after the sizeof above, right?
quoted
 
+	if (verbose)
+		strncat(script, " -v", sizeof(script));
+
Seemed simple enough, but gcc knows better, I'm removing this one:

    tests/builtin-test.c:586:26: error: the value of the size argument in 'strncat' is too large, might lead to a buffer overflow [-Werror,-Wstrncat-size]
                    strncat(script, " -v", sizeof(script));
                                           ^~~~~~~~~~~~~~
    tests/builtin-test.c:586:26: note: change the argument to be the free space in the destination buffer minus the terminating null byte
                    strncat(script, " -v", sizeof(script));
                                           ^~~~~~~~~~~~~~
                                           sizeof(script) - strlen(script) - 1
    1 error generated.
    make[3]: *** [/git/perf-5.13.0-rc4/tools/build/Makefile.build:139: tests] Error 2
  77    31.98 ubuntu:21.04                  : FAIL gcc version 10.3.0 (Ubuntu 10.3.0-1ubuntu1)
    tests/builtin-test.c:586:26: error: the value of the size argument in 'strncat' is too large, might lead to a buffer overflow [-Werror,-Wstrncat-size]
                    strncat(script, " -v", sizeof(script));
                                           ^~~~~~~~~~~~~~
    tests/builtin-test.c:586:26: note: change the argument to be the free space in the destination buffer minus the terminating null byte
                    strncat(script, " -v", sizeof(script));
                                           ^~~~~~~~~~~~~~
                                           sizeof(script) - strlen(script) - 1
    1 error generated.
    make[3]: *** [/git/perf-5.13.0-rc4/tools/build/Makefile.build:139: tests] Error 2

quoted
 	err = system(script);
 	if (!err)
 		return TEST_OK;
-- 
2.32.0.288.g62a8d224e6-goog
-- 

- Arnaldo
-- 

- Arnaldo

Re: [PATCH 2/4] perf test: Pass the verbose option to shell tests

From: Ian Rogers <irogers@google.com>
Date: 2021-06-19 04:45:25

On Fri, Jun 18, 2021 at 9:49 AM Arnaldo Carvalho de Melo
[off-list ref] wrote:
Em Thu, Jun 17, 2021 at 04:19:06PM -0300, Arnaldo Carvalho de Melo escreveu:
quoted
Em Thu, Jun 17, 2021 at 11:42:14AM -0700, Ian Rogers escreveu:
quoted
Having a verbose option will allow shell tests to provide extra failure
details when the fail or skip.

Thanks, applied to perf/core.

- Arnaldo
quoted
Signed-off-by: Ian Rogers <irogers@google.com>
---
 tools/perf/tests/builtin-test.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index cbbfe48ab802..a8160b1684de 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -577,11 +577,14 @@ struct shell_test {
 static int shell_test__run(struct test *test, int subdir __maybe_unused)
 {
    int err;
-   char script[PATH_MAX];
+   char script[PATH_MAX + 3];
    struct shell_test *st = test->priv;

    path__join(script, sizeof(script), st->dir, st->file);
probably you need to add a  '- 3' after the sizeof above, right?
Either way is fine, but -3 is ok with me.
quoted
quoted
+   if (verbose)
+           strncat(script, " -v", sizeof(script));
+
Seemed simple enough, but gcc knows better, I'm removing this one:

    tests/builtin-test.c:586:26: error: the value of the size argument in 'strncat' is too large, might lead to a buffer overflow [-Werror,-Wstrncat-size]
                    strncat(script, " -v", sizeof(script));
                                           ^~~~~~~~~~~~~~
    tests/builtin-test.c:586:26: note: change the argument to be the free space in the destination buffer minus the terminating null byte
                    strncat(script, " -v", sizeof(script));
                                           ^~~~~~~~~~~~~~
                                           sizeof(script) - strlen(script) - 1
    1 error generated.
    make[3]: *** [/git/perf-5.13.0-rc4/tools/build/Makefile.build:139: tests] Error 2
  77    31.98 ubuntu:21.04                  : FAIL gcc version 10.3.0 (Ubuntu 10.3.0-1ubuntu1)
    tests/builtin-test.c:586:26: error: the value of the size argument in 'strncat' is too large, might lead to a buffer overflow [-Werror,-Wstrncat-size]
                    strncat(script, " -v", sizeof(script));
                                           ^~~~~~~~~~~~~~
    tests/builtin-test.c:586:26: note: change the argument to be the free space in the destination buffer minus the terminating null byte
                    strncat(script, " -v", sizeof(script));
                                           ^~~~~~~~~~~~~~
                                           sizeof(script) - strlen(script) - 1
    1 error generated.
    make[3]: *** [/git/perf-5.13.0-rc4/tools/build/Makefile.build:139: tests] Error 2
Thanks gcc :-) Do you want me to resend the patch?

Ian
quoted
quoted
    err = system(script);
    if (!err)
            return TEST_OK;
--
2.32.0.288.g62a8d224e6-goog
--

- Arnaldo
--

- Arnaldo

Re: [PATCH 2/4] perf test: Pass the verbose option to shell tests

From: Arnaldo Carvalho de Melo <acme@kernel.org>
Date: 2021-06-19 13:04:43

Em Fri, Jun 18, 2021 at 09:45:05PM -0700, Ian Rogers escreveu:
On Fri, Jun 18, 2021 at 9:49 AM Arnaldo Carvalho de Melo
[off-list ref] wrote:
quoted
Em Thu, Jun 17, 2021 at 04:19:06PM -0300, Arnaldo Carvalho de Melo escreveu:
quoted
Em Thu, Jun 17, 2021 at 11:42:14AM -0700, Ian Rogers escreveu:
quoted
Having a verbose option will allow shell tests to provide extra failure
details when the fail or skip.

Thanks, applied to perf/core.

- Arnaldo
quoted
Signed-off-by: Ian Rogers <irogers@google.com>
---
 tools/perf/tests/builtin-test.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index cbbfe48ab802..a8160b1684de 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -577,11 +577,14 @@ struct shell_test {
 static int shell_test__run(struct test *test, int subdir __maybe_unused)
 {
    int err;
-   char script[PATH_MAX];
+   char script[PATH_MAX + 3];
    struct shell_test *st = test->priv;

    path__join(script, sizeof(script), st->dir, st->file);
probably you need to add a  '- 3' after the sizeof above, right?
Either way is fine, but -3 is ok with me.
quoted
quoted
quoted
+   if (verbose)
+           strncat(script, " -v", sizeof(script));
+
Seemed simple enough, but gcc knows better, I'm removing this one:

    tests/builtin-test.c:586:26: error: the value of the size argument in 'strncat' is too large, might lead to a buffer overflow [-Werror,-Wstrncat-size]
                    strncat(script, " -v", sizeof(script));
                                           ^~~~~~~~~~~~~~
    tests/builtin-test.c:586:26: note: change the argument to be the free space in the destination buffer minus the terminating null byte
                    strncat(script, " -v", sizeof(script));
                                           ^~~~~~~~~~~~~~
                                           sizeof(script) - strlen(script) - 1
    1 error generated.
    make[3]: *** [/git/perf-5.13.0-rc4/tools/build/Makefile.build:139: tests] Error 2
  77    31.98 ubuntu:21.04                  : FAIL gcc version 10.3.0 (Ubuntu 10.3.0-1ubuntu1)
    tests/builtin-test.c:586:26: error: the value of the size argument in 'strncat' is too large, might lead to a buffer overflow [-Werror,-Wstrncat-size]
                    strncat(script, " -v", sizeof(script));
                                           ^~~~~~~~~~~~~~
    tests/builtin-test.c:586:26: note: change the argument to be the free space in the destination buffer minus the terminating null byte
                    strncat(script, " -v", sizeof(script));
                                           ^~~~~~~~~~~~~~
                                           sizeof(script) - strlen(script) - 1
    1 error generated.
    make[3]: *** [/git/perf-5.13.0-rc4/tools/build/Makefile.build:139: tests] Error 2
Thanks gcc :-) Do you want me to resend the patch?
In such cases please do,

- Arnaldo

Re: [PATCH 1/4] perf test: Fix non-bash issue with stat bpf counters

From: Stephen Rothwell <hidden>
Date: 2021-06-20 21:55:36

Hi Ian,

On Thu, 17 Jun 2021 11:42:13 -0700 Ian Rogers [off-list ref] wrote:
$(( .. )) is a bash feature but the test's interpreter is !/bin/sh,
switch the code to use expr.
The $(( .. )) syntax is specified in POSIX (see
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_04),
so unless this caused an actual problem, this change is unnecessary.

-- 
Cheers,
Stephen Rothwell

Re: [PATCH 1/4] perf test: Fix non-bash issue with stat bpf counters

From: Ian Rogers <irogers@google.com>
Date: 2021-06-21 21:31:08

On Sun, Jun 20, 2021 at 2:55 PM Stephen Rothwell [off-list ref] wrote:
Hi Ian,

On Thu, 17 Jun 2021 11:42:13 -0700 Ian Rogers [off-list ref] wrote:
quoted
$(( .. )) is a bash feature but the test's interpreter is !/bin/sh,
switch the code to use expr.
The $(( .. )) syntax is specified in POSIX (see
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_04),
so unless this caused an actual problem, this change is unnecessary.
Agreed. The issue I was seeing was:

./tests/shell/stat_bpf_counters.sh: line 14: <not + <not / 10 : syntax
error: operand expected (error token is "<not + <not / 10 ")

but that syntax error is caused by running the test within a
hypervisor. I'll resend the patch set with this one dropped.

Thanks,
Ian
--
Cheers,
Stephen Rothwell
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help