Hi Alex,

Sent v5 version of patches with changes as per my last mail.
Also, attached "summary.txt" of rust testing with arm64 target, for
your reference.

Regards,
Vinay

On Thu, May 27, 2021 at 2:13 AM Vinay Kumar [off-list ref] wrote:

Hi Alex,

quoted

quoted

Hardcoding the ip port this way is problematic and prone to conflicts. What is -monitor used for?

When we were testing outside oe-selftest framework ( booting image
manually and trigger testing),
the "remote-test-server" used to give connection issues. However,
right now without this option the testing is working without any
issue. So removed from qemuboot.

quoted

quoted

What does -j 40 do? Keep in mind that qemu has only 4 virtual cpu cores.

This one passed to speed up the testing while using the "do_check"
method. So I removed parallel builds.

quoted

quoted

What happens if the test executed via runCmd fails? Specifically, will the failure be properly reported?

While executing testing, if there any testcase failures testing gets
terminated with below message,
"command did not execute successfully:"

So we are passing "--no-fail-fast" as a test argument, which wont exit
the shell and the testing continues even if there are failures.

quoted

quoted

I'm wondering because the output of the test is redirected to summary.txt (which is then left unused), and stderr is discarded altogether.

This is because I only capture the test results as much as possible.
Now modified to consider stderr also.

quoted

quoted

Will t_thread be stopped if runCmd fails?

Now changed the implementation to discard thread usage. Even in case
of any failure the test used to end after timeout set in "SShcontrol"
if the timeout is less (60 seconds) then some of the tests used to get missed.

So instead of using copy_to and run from sshcontrol.py, I tried
copying the remote-test-server to the image and  executing in the
background as below,

#Copy remote-test-server through scp and execute through background ssh
sshargs = '-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no
-o LogLevel=ERROR'
cmd = "scp %s %s/build/x86_64-unknown-linux-gnu/stage2-tools-bin/remote-test-server
root@%s:~/ ;" % (sshargs, builddir, qemu.ip)
testcommand = '~/remote-test-server -v remote'
cmd = cmd + "ssh %s -f root@%s \"%s\";" % (sshargs, qemu.ip, testcommand)

And kill "remote-test-server" through ssh once testing is complete
killcommand = "kill -9 \$(ps | grep remote-test-server |  grep -v
\"grep\" | awk '{print \$1}')"
cmd = cmd + "ssh %s root@%s \"%s\";" % (sshargs, qemu.ip, killcommand)

Testing in progress will send the v5 version patches.

Regards,
Vinay

On Tue, May 25, 2021 at 1:29 AM Alexander Kanavin
[off-list ref] wrote:

quoted

On Mon, 24 May 2021 at 19:16, Vinay Kumar [off-list ref] wrote:

quoted

Build remote-test-server copy to image and execute
remotely through background ssh.

Thanks, I do not have any further comments about the overall structure, the below are more specific concerns.

quoted

+               with runqemu("core-image-minimal", runqemuparams = "nographic", qemuparams= " -monitor \
+                        telnet:127.0.0.1:1234,server,nowait -serial mon:stdio -serial null") as qemu:

Hardcoding the ip port this way is problematic and prone to conflicts. What is -monitor used for?

quoted

+                       cmd = cmd + " cd %s;  python3 src/bootstrap/bootstrap.py -j 40 --verbose test %s --target %s > summary.txt 2> /dev/null " % (builddir, testargs, targetsys)

What does -j 40 do? Keep in mind that qemu has only 4 virtual cpu cores.

quoted

+                       runCmd(cmd)

What happens if the test executed via runCmd fails? Specifically, will the failure be properly reported?

I'm wondering because the output of the test is redirected to summary.txt (which is then left unused), and stderr is discarded altogether.

quoted

+                       # Stop ssh background execution
+                       t_thread.stop()

Will t_thread be stopped if runCmd fails?

Alex