Here's a non-RFC version of the smoke testing for Git series. It's
like v2 of the RFC aside from a s/sever/server/ typo noted by Junio:
Ævar Arnfjörð Bjarmason (2):
tests: Infrastructure for Git smoke testing
t/README: Document the Smoke testing
t/Makefile | 35 ++++++++++++++++++++++++++++++-
t/README | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
t/harness | 21 +++++++++++++++++++
3 files changed, 121 insertions(+), 1 deletions(-)
create mode 100755 t/harness
Add the capability to send smoke reports from the Git test suite.
Currently we only notice bugs in the test suite when it's run
manually. Bugs in Git that only occur on obscure platforms or setups
that the core developers aren't using can thus go unnoticed.
This series aims to change that. With it anyone that's interested in
avoiding bitrot in Git can volunteer to run a smoke tester. A smoke
tester periodically compiles the latest version of Git, runs the test
suite, and submits a report to a central server indicating how the
test run went.
A smoke tester might run something like this in cron:
#!/bin/sh
cd ~/g/git
git fetch
for branch in maint master next pu; do
git checkout origin/$i &&
make clean all &&
cd t &&
make smoke_report
done
The smoker might want to compile git with non-default flags, include
bisecting functionality or run the tests under valgrind. Doing that is
outside the scope of this patch, this just adds a report submission
mechanism. But including a canonical smoke runner is something we'll
want to include eventually.
What this does now is add smoke and smoke_report targets to t/Makefile
(this example only uses a few tests for demonstration):
$ make clean smoke
rm -f -r 'trash directory'.* test-results
rm -f t????/cvsroot/CVSROOT/?*
rm -f -r valgrind/bin
rm -f .prove
perl ./harness --git-version="1.7.2.1.173.gc9b40" \
--no-verbose \
--archive="test-results/git-smoke.tar.gz" \
t0000-basic.sh t0001-init.sh t0002-gitfile.sh t0003-attributes.sh t0004-unwritable.sh t0005-signals.sh t0006-date.sh
t0000-basic.sh ....... ok
t0001-init.sh ........ ok
t0002-gitfile.sh ..... ok
t0003-attributes.sh .. ok
t0004-unwritable.sh .. ok
t0005-signals.sh ..... ok
t0006-date.sh ........ ok
All tests successful.
Test Summary Report
-------------------
t0000-basic.sh (Wstat: 0 Tests: 46 Failed: 0)
TODO passed: 5
Files=7, Tests=134, 3 wallclock secs ( 0.06 usr 0.05 sys + 0.23 cusr 1.33 csys = 1.67 CPU)
Result: PASS
TAP Archive created at /home/avar/g/git/t/test-results/git-smoke.tar.gz
The smoke target uses TAP::Harness::Archive to aggregate the test
results into a tarball. The tarball contains two things, the output of
every test file that was run, and a metadata file:
Tarball contents:
$ tar xzvf git-smoke.tar.gz
t0004-unwritable.sh
t0001-init.sh
t0002-gitfile.sh
t0005-signals.sh
t0000-basic.sh
t0003-attributes.sh
t0006-date.sh
meta.yml
A test report:
$ cat t0005-signals.sh
ok 1 - sigchain works
# passed all 1 test(s)
1..1
A metadata file:
---
extra_properties:
file_attributes:
-
description: t0000-basic.sh
end_time: 1280437324.61398
start_time: 1280437324.22186
-
description: t0001-init.sh
end_time: 1280437325.12346
start_time: 1280437324.62393
-
description: t0002-gitfile.sh
end_time: 1280437325.29428
start_time: 1280437325.13646
-
description: t0003-attributes.sh
end_time: 1280437325.59678
start_time: 1280437325.30565
-
description: t0004-unwritable.sh
end_time: 1280437325.77376
start_time: 1280437325.61003
-
description: t0005-signals.sh
end_time: 1280437325.85426
start_time: 1280437325.78727
-
description: t0006-date.sh
end_time: 1280437326.2362
start_time: 1280437325.86768
file_order:
- t0000-basic.sh
- t0001-init.sh
- t0002-gitfile.sh
- t0003-attributes.sh
- t0004-unwritable.sh
- t0005-signals.sh
- t0006-date.sh
start_time: 1280437324
stop_time: 1280437326
The "extra_properties" hash is where we'll stick Git-specific info,
like whether Git was compiled with gettext or the fallback regex
engine, and what branch we're compiling. Currently no metadata like
this is included.
The entire tarball is then submitted to a central smokebox at
smoke.git.nix.is. This is done with curl(1) via the "smoke_report"
target:
$ make smoke_report
curl \
-H "Expect: " \
-F project=Git \
-F architecture=x86_64 \
-F platform=Linux \
-F revision="1.7.2.1.173.gc9b40" \
-F report_file=@test-results/git-smoke.tar.gz \
http://smoke.git.nix.is/app/projects/process_add_report/1 \
| grep -v ^Redirecting
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 117k 100 63 100 117k 3 6430 0:00:21 0:00:18 0:00:03 0
Reported #8 added.
Reports are then made available on the smokebox via a web interface:
http://smoke.git.nix.is/app/projects/smoke_reports/1
The smoke reports are also mirrored to a Git repository hosted on
GitHub:
http://github.com/gitsmoke/smoke-reports
The Smolder SQLite database that contains metadata about the reports
is also made available:
http://github.com/gitsmoke/smoke-database
Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---
t/Makefile | 35 ++++++++++++++++++++++++++++++++++-
t/harness | 21 +++++++++++++++++++++
2 files changed, 55 insertions(+), 1 deletions(-)
create mode 100755 t/harness
Git now has a smoke testing service at http://smoke.git.nix.is that
anyone can send reports to. Change the t/README file to mention this.
Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---
t/README | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 files changed, 66 insertions(+), 0 deletions(-)
@@ -574,3 +574,69 @@ Then, at the top-level: That'll generate a detailed cover report in the "cover_db_html" directory, which you can then copy to a webserver, or inspect locally in a browser.++Smoke testing+-------------++The Git test suite has support for smoke testing. Smoke testing is+when you submit the results of a test run to a central server for+analysis and aggregation.++Running a smoke tester is an easy and valuable way of contributing to+Git development, particularly if you have access to an uncommon OS on+obscure hardware.++After building Git you can generate a smoke report like this in the+"t" directory:++ make clean smoke++You can also pass arguments via the environment. This should make it+faster:++ GIT_TEST_OPTS='--root=/dev/shm' TEST_JOBS=10 make clean smoke++The "smoke" target will run the Git test suite with Perl's+"TAP::Harness" module, and package up the results in a .tar.gz archive+with "TAP::Harness::Archive". The former is included with Perl v5.10.1+or later, but you'll need to install the latter from the CPAN. See the+"Test coverage" section above for how you might do that.++Once the "smoke" target finishes you'll see a message like this:++ TAP Archive created at <path to git>/t/test-results/git-smoke.tar.gz++To upload the smoke report you need to have curl(1) installed, then+do:++ make smoke_report++To upload the report anonymously. Hopefully that'll return something+like "Reported #7 added.".++If you're going to be uploading reports frequently please request a+user account by E-Mailing gitsmoke@v.nix.is. Once you have a username+and password you'll be able to do:++ SMOKE_USERNAME=<username> SMOKE_PASSWORD=<password> make smoke_report++Once the report is uploaded it'll be made available at+http://smoke.git.nix.is, here's an overview of Recent Smoke Reports+for Git:++ http://smoke.git.nix.is/app/projects/smoke_reports/1++The reports will also be mirrored to GitHub every few hours:++ http://github.com/gitsmoke/smoke-reports++The Smolder SQLite database is also mirrored and made available for+download:++ http://github.com/gitsmoke/smoke-database++Note that the database includes hashed (with crypt()) user passwords+and E-Mail addresses. Don't use a valuable password for the smoke+service if you have an account, or an E-Mail address you don't want to+be publicly known. The user accounts are just meant to be convenient+labels, they're not meant to be secure.
Heya,
On Mon, Aug 2, 2010 at 15:24, Ævar Arnfjörð Bjarmason [off-list ref] wrote:
Git now has a smoke testing service at http://smoke.git.nix.is that
anyone can send reports to. Change the t/README file to mention this.
Can't we run this somewhere a little less "shabby"? I'd hate to
introduce Yet Another Official Git Site (I'm really glad for example
that the Git Wiki is now on kernel.org). Is it not possible to move
this to git-scm.com, kernel.org, or something like that?
--
Cheers,
Sverre Rabbelier
From: A Large Angry SCM <hidden> Date: 2016-06-15 22:49:14
On 08/02/2010 04:54 PM, Sverre Rabbelier wrote:
Heya,
On Mon, Aug 2, 2010 at 15:24, Ævar Arnfjörð Bjarmason[off-list ref] wrote:
quoted
Git now has a smoke testing service at http://smoke.git.nix.is that
anyone can send reports to. Change the t/README file to mention this.
Can't we run this somewhere a little less "shabby"? I'd hate to
introduce Yet Another Official Git Site (I'm really glad for example
that the Git Wiki is now on kernel.org). Is it not possible to move
this to git-scm.com, kernel.org, or something like that?
And what's on the site tells us almost nothing about the tested systems.
On Mon, Aug 2, 2010 at 21:04, A Large Angry SCM [off-list ref] wrote:
On 08/02/2010 04:54 PM, Sverre Rabbelier wrote:
quoted
Heya,
On Mon, Aug 2, 2010 at 15:24, Ævar Arnfjörð Bjarmason[off-list ref]
wrote:
quoted
Git now has a smoke testing service at http://smoke.git.nix.is that
anyone can send reports to. Change the t/README file to mention this.
Can't we run this somewhere a little less "shabby"? I'd hate to
introduce Yet Another Official Git Site (I'm really glad for example
that the Git Wiki is now on kernel.org). Is it not possible to move
this to git-scm.com, kernel.org, or something like that?
Do you not like the domain, that the domain's A record points to a
Linode box, or both?
I don't mind where a smoke server gets hosted, I just want it to be
there. But someone's going to have to host it and maintain it.
Unless someone else wants to do that it's going to have to either be
at a box I control, or someone else will need to give me shell access.
Of course just changing the domain would be trivial, just a matter of
someone setting up an A record somewhere, or delegating a zone to me.
An alternative would be to change how this works, e.g. making
submissions go by E-Mail to a public list, or having smokers push them
to public Git repositories which would be registered somewhere.
But that'd require more development and setup work. The current system
isn't optimal, but it works. It can always me moved somewhere else
later if smoke testing for Git takes off.
And what's on the site tells us almost nothing about the tested systems.
Heya,
[+Warthog9]
On Mon, Aug 2, 2010 at 17:00, Ævar Arnfjörð Bjarmason [off-list ref] wrote:
Do you not like the domain, that the domain's A record points to a
Linode box, or both?
The former.
I don't mind where a smoke server gets hosted, I just want it to be
there. But someone's going to have to host it and maintain it.
Agreed.
Of course just changing the domain would be trivial, just a matter of
someone setting up an A record somewhere, or delegating a zone to me.
John, is it possible to have smoke.git.kernel.org (or such a domain)
either point at a server Ævar controls, or to give him a shell account
there? The point is to have http://smoke.git.nix.is/ hosted on a more
"Gitty" domain.
An alternative would be to change how this works, e.g. making
submissions go by E-Mail to a public list, or having smokers push them
to public Git repositories which would be registered somewhere.
I like the idea of having it be a public Git repo since that would
probably make it easier to do maintenance on the repo (e.g., revert
"vandalism"), but there still needs to be a frontend of sorts
somewhere :).
--
Cheers,
Sverre Rabbelier
Operating system (i.e., distro). Precise CPU architecture. Number of
CPUs. Available memory. Which tests were skipped and which had expected
failures. Configuration used to build git.
(I understand that requiring all that information could prevent people
from running smoke servers, so I don’t mean to say it all should be
required.)
The only information available there is that the tested system was some
Linux on an x86_64 and the particular git version tested. My complaint
is that "Linux" is _way_ too generic to be useful when trying to get an
idea of where things are known to work and where they are known not to work.
On Mon, Aug 2, 2010 at 22:05, Sverre Rabbelier [off-list ref] wrote:
On Mon, Aug 2, 2010 at 17:00, Ævar Arnfjörð Bjarmason [off-list ref] wrote:
quoted
Do you not like the domain, that the domain's A record points to a
Linode box, or both?
The former.
quoted
I don't mind where a smoke server gets hosted, I just want it to be
there. But someone's going to have to host it and maintain it.
Agreed.
quoted
Of course just changing the domain would be trivial, just a matter of
someone setting up an A record somewhere, or delegating a zone to me.
John, is it possible to have smoke.git.kernel.org (or such a domain)
either point at a server Ævar controls, or to give him a shell account
there? The point is to have http://smoke.git.nix.is/ hosted on a more
"Gitty" domain.
Note that that may be a bit premature, since Junio hasn't commented on
whether he wants this at all yet.
quoted
An alternative would be to change how this works, e.g. making
submissions go by E-Mail to a public list, or having smokers push them
to public Git repositories which would be registered somewhere.
I like the idea of having it be a public Git repo since that would
probably make it easier to do maintenance on the repo (e.g., revert
"vandalism"), but there still needs to be a frontend of sorts
somewhere :).
Note that all the data is already in Git[1] it just isn't used as the
submission mechanism.
1. http://github.com/gitsmoke
On Mon, Aug 2, 2010 at 4:24 PM, Ævar Arnfjörð Bjarmason
[off-list ref] wrote:
The smoker might want to compile git with non-default flags, include
bisecting functionality or run the tests under valgrind. Doing that is
outside the scope of this patch, this just adds a report submission
mechanism. But including a canonical smoke runner is something we'll
want to include eventually.
You might want to play with gitbuilder:
http://github.com/apenwarr/gitbuilder
It can bisect automatically and it looks like it could integrate
pretty nicely with this system. (If you're running a gitbuilder, just
have it submit the smoke test with each build it makes. gitbuilder
will automatically bisect if 'make test' fails, so eventually you'll
submit smoke tests that *don't* fail.)
Have fun,
Avery