On Mon, Oct 09, 2017 at 04:17:31PM +1100, Dave Chinner wrote:

On Sun, Oct 08, 2017 at 10:25:10PM -0400, Josef Bacik wrote:

quoted

On Mon, Oct 09, 2017 at 11:51:37AM +1100, Dave Chinner wrote:

quoted

On Fri, Oct 06, 2017 at 05:09:57PM -0400, Josef Bacik wrote:

quoted

Hello,

One thing that comes up a lot every LSF is the fact that we have no general way
that we do performance testing.  Every fs developer has a set of scripts or
things that they run with varying degrees of consistency, but nothing central
that we all use.  I for one am getting tired of finding regressions when we are
deploying new kernels internally, so I wired this thing up to try and address
this need.

We all hate convoluted setups, the more brain power we have to put in to setting
something up the less likely we are to use it, so I took the xfstests approach
of making it relatively simple to get running and relatively easy to add new
tests.  For right now the only thing this framework does is run fio scripts.  I
chose fio because it already gathers loads of performance data about it's runs.
We have everything we need there, latency, bandwidth, cpu time, and all broken
down by reads, writes, and trims.  I figure most of us are familiar enough with
fio and how it works to make it relatively easy to add new tests to the
framework.

I've posted my code up on github, you can get it here

https://github.com/josefbacik/fsperf

All (well most) of the results from fio are stored in a local sqlite database.
Right now the comparison stuff is very crude, it simply checks against the
previous run and it only checks a few of the keys by default.  You can check
latency if you want, but while writing this stuff up it seemed that latency was
too variable from run to run to be useful in a "did my thing regress or improve"
sort of way.

The configuration is brain dead simple, the README has examples.  All you need
to do is make your local.cfg, run ./setup and then run ./fsperf and you are good
to go.

Why re-invent the test infrastructure? Why not just make it a
tests/perf subdir in fstests?

Probably should have led with that shouldn't I have?  There's nothing keeping me
from doing it, but I didn't want to try and shoehorn in a python thing into
fstests.  I need python to do the sqlite and the json parsing to dump into the
sqlite database.

Now if you (and others) are not opposed to this being dropped into tests/perf
then I'll work that up.  But it's definitely going to need to be done in python.
I know you yourself have said you aren't opposed to using python in the past, so
if that's still the case then I can definitely wire it all up.

I have no problems with people using python for stuff like this but,
OTOH, I'm not the fstests maintainer anymore :P

I have no problem either if python is really needed, after all this is a
very useful infrastructure improvement. But the python version problem
brought up by Ted made me a bit nervous, we need to work that round
carefully.

OTOH, I'm just curious, what is the specific reason that python is a
hard requirement? If we can use perl, that'll be much easier for
fstests.

BTW, opinions from key fs developers/fstests users, like you, are also
very important and welcomed :)

Thanks,
Eryu

quoted

quoted

quoted

The plan is to add lots of workloads as we discover regressions and such.  We
don't want anything that takes too long to run otherwise people won't run this,
so the existing tests don't take much longer than a few minutes each.  I will be
adding some more comparison options so you can compare against averages of all
previous runs and such.

Yup, that fits exactly into what fstests is for... :P

Integrating into fstests means it will be immediately available to
all fs developers, it'll run on everything that everyone already has
setup for filesystem testing, and it will have familiar mkfs/mount
option setup behaviour so there's no new hoops for everyone to jump
through to run it...

TBF I specifically made it as easy as possible because I know we all hate trying
to learn new shit.

Yeah, it's also hard to get people to change their workflows to add
a whole new test harness into them. It's easy if it's just a new
command to an existing workflow :P

quoted

I figured this was different enough to warrant a separate
project, especially since I'm going to add block device jobs so Jens can test
block layer things.  If we all agree we'd rather see this in fstests then I'm
happy to do that too.  Thanks,

I'm not fussed either way - it's a good discussion to have, though.

If I want to add tests (e.g. my time-honoured fsmark tests), where
should I send patches?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com
--
To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html