Automated performance testing system was Re: Text form for STP tests

Linux-MM fellows,

I've been talking to Cliff about the need for a set of benchmarks,
covering as many different workloads as possible, for developers to have a 
better notion of impact on performance changes. 

Usually when one does a change which affects performance, he/she runs one 
or two benchmarks with a limited amount of hardware configurations.
This is a very painful, boring and time consuming process, which can 
result in misinterpretation and/or limited understading of the results 
of such changes.

It is important to automate such process, with a set of benchmarks 
covering as wide as possible range of workloads, running on common 
and most used hardware variations.

OSDL's STP provides the base framework for this.

Cliff mentioned an internal tool they are developing for this purpose, 
based on XML-like configuration files. 

I have suggested him a set of benchmarks (available on STP right now, 
we want to add other benchmarks there whenever necessary) and a set of 
CPU/memory variations.

Quoting myself:

"here is a list of a wide enough 
group of performance tests, for a start:

bonnie++
reaim (default, new_fserver, shared)
dbench_long
kernbench
tiobench

Each of these running one the following combinations:

1CPU, 2CPU, 4CPU, 8CPU (4 variants).

total memory, half memory, a quarter of total memory (3 variants).

Thats 12 results for each benchmark."

Obviously this set of benchmarks is limited (an example set), they are 
relatively similar, but it is a start. 
In the future we ough to have several sets of benchmarks.

We also need a way to easily visualize such results, which is the next
step of the project. x,y graphics are the best way for most tests 
I believe.

Construction of such automated testing infrastructure will improve our 
capabilities giving better notion of impact while decreasing 
wasted time and wasted efforts.

Of course this needs to be a collaborative effort, I expect others
interested to help, comment and get involved.

Follows my private discussion with Cliff up till now, for reference.

On Sun, Nov 28, 2004 at 12:17:02PM -0800, Cliff White wrote:

quoted

Hi Cliff,

On Wed, Nov 24, 2004 at 10:22:57AM -0800, Cliff White wrote:

quoted

Marcelo, we are working on a better version of 
this as I mentioned, plus a command line tool.

Great :)

quoted

The current file is almost XML, and some
of the data is internal to our database. If you
can indicate what tests you want to run on specific
sizes of machines,  we can supply all the numbers. 

It the XML is too much, just send me a list of what you
want, and we can go from there.

That looks fine to me - here is a list of a wide enough 
group of performance tests, for a start:

bonnie++
reaim (default, new_fserver, shared)
dbench_long
kernbench
tiobench

Each of these running one the following combinations:

1CPU, 2CPU, 4CPU, 8CPU (4 variants).

total memory, half memory, a quarter of total memory (3 variants).

Thats 12 results for each benchmark. 

I can get this setup monday, ( long weekend ) 

quoted

We need to make those results visually easy to read/interpret, 
but thats another story which can be done independantly.
A x,y graph is the easier way to visualize - thats the next step.

Great, appreciate any input on this. 

quoted

quoted

We have an automated tool that does a string match against
kernel.org patch names, if you have a specific regexp, we
can tie an automatic test submit to that. 

The text file just specifies test details, kernel patch info
is supplied in a separate step, so one text file can
be re-used.

OK - great. We want to run the testgroup on baseline kernel 
and on baseline+modification, of course.

Can we move this discussion to linux-mm? I'm sure others are interested
as well.

Sure, would you start a thread? I'm on that list already.
cliffw

quoted

Many thanks, this will certainly improve out performance testing capabilities.

quoted

------------Current-------------------
<config >
    <stp2-000 host="77">
        <aio-stress testID="82">
            <distro>4</distro>
            <run>
                <lilo>profile=2</lilo>
                <opt pID="-F" val="1" />
                <opt pID="-s" val="4g" />
            </run>
        </aio-stress>
    </stp2-000>
</config>
----------------------------------

We've changed the format some, but the tools to use
it are still being fixed up. New format (so far)
will look like this: ( the 'test name' and 'host name' bits allow
for multiple runs of the same test/host )

--------------------------------

<config>
  <host name="cpu1" id="14">
    <test name="dummy_test" id="82">
      <distro id="4" />
      <run>
        <lilo>profile=2</lilo>
        <param name="-o">25</param>
        <param name="-z">foobar</param>
      </run>
      <run>
        <lilo>profile=2</lilo>
        <param name="-o">25</param>
        <param name="-z">foobar</param>
      </run>
    </test>
  </host>
 <host name="cpu2" id="15">
    <test name="dummy_test" id="82">
    <distro id="4" />
      <run>
        <lilo>profile=2</lilo>
        <param name="-o">25</param>
        <param name="-z">foobar</param>
      </run>
    </test>
  </host>
</config>
----------------------------------

cliffw

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

I've been talking to Cliff about the need for a set of benchmarks,
covering as many different workloads as possible, for developers to have a 
better notion of impact on performance changes. 

Usually when one does a change which affects performance, he/she runs one 
or two benchmarks with a limited amount of hardware configurations.
This is a very painful, boring and time consuming process, which can 
result in misinterpretation and/or limited understading of the results 
of such changes.

It is important to automate such process, with a set of benchmarks 
covering as wide as possible range of workloads, running on common 
and most used hardware variations.

OSDL's STP provides the base framework for this.

Cliff mentioned an internal tool they are developing for this purpose, 
based on XML-like configuration files. 

I have suggested him a set of benchmarks (available on STP right now, 
we want to add other benchmarks there whenever necessary) and a set of 
CPU/memory variations.

Sounds like a good plan in general, by why on earth would you want to do
it in XML? Personally I'm not that much into masochism. A simple text
control file is perfectly sufficient (and yes, we do this internally).

M.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

On Tue, 30 Nov 2004 09:00:59 -0800
"Martin J. Bligh" [off-list ref] wrote:

quoted

I've been talking to Cliff about the need for a set of benchmarks,
covering as many different workloads as possible, for developers to have a 
better notion of impact on performance changes. 

Usually when one does a change which affects performance, he/she runs one 
or two benchmarks with a limited amount of hardware configurations.
This is a very painful, boring and time consuming process, which can 
result in misinterpretation and/or limited understading of the results 
of such changes.

It is important to automate such process, with a set of benchmarks 
covering as wide as possible range of workloads, running on common 
and most used hardware variations.

OSDL's STP provides the base framework for this.

Cliff mentioned an internal tool they are developing for this purpose, 
based on XML-like configuration files. 

I have suggested him a set of benchmarks (available on STP right now, 
we want to add other benchmarks there whenever necessary) and a set of 
CPU/memory variations.

Sounds like a good plan in general, by why on earth would you want to do
it in XML? Personally I'm not that much into masochism. A simple text
control file is perfectly sufficient (and yes, we do this internally).

True, very true. What i was showing Marcelo was what we do internally, which
was not desiged for humans. What we're working on currently _is designed for humans,
and not XML.

I'm making up this AM a control file for STP for the tests marcelo requested, and
we won't be asking humans to do XML, no. 

Martin, if you could share any of your internal goop ( and save me work ) it would be 
great. If you all have a text format that large numbers of people find sensible, i'd love it.
So far, it's just been me and the robots. 

( btw, it's XML because XML::Simple just seemed so....simple. I was young then. ) 
cliffw

M.

-- 
The church is near, but the road is icy.
The bar is far, but i will walk carefully. - Russian proverb
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>