Re: serious performance issues with images, audio files, and other "non-code" data

On Tue, May 18, 2010 at 02:50:16PM -0400, John wrote:

I just compiled the latest git. It got worse!!

I think Sverre is right that this is simply that --aggressive got more
so in the last few versions. But do note that aggressive implies that we
should pack from scratch, not reusing previously found deltas (or
accepting that we didn't find deltas previously).

So you might want "git gc --aggressive" the _first_ time you pack, or
possibly even very occasionally. But if you are packing every day, you
should just use "git gc", which will run much more quickly (and would
probably have acceptable behavior even without the -delta attribute, as
it would only have to look at _new_ objects).

It will have to write the whole 200M packfile out each time, though.
From your timings that looks to take about 50 seconds or so (just
looking at the difference between wall clock time and CPU time, which is
presumably spent in I/O).

Packing nightly won't hurt, but is perhaps excessive. It sounds like you
actually have a fairly normal workload.

quoted

How big is your packed repo? Given the pattern you describe below, I am

[...]
The packed .git dir is 203 MB. Yes, we make frequent changes to these
files, and push/pull frequently as well. Just a normal development
pattern, though. It's all manually done -- i.e., there's no automated
bot doing excessive git operations.

OK, that is not very big. Once packed, you really should not see
performance issues.

culprit. Yes, network bandwidth is always a factor, but I guess my
expectation is that git shouldn't transfer too much more info than
the amount of recent changes. For example, if we change 10 files for
a total of 10MB, then my admittedly naive expectation is that git
will send that 10MB of changes, plus some small constant amount of
meta info... not the whole repo every time. No?

Your assumption is correct. Git should transmit at _worst_ 10MB in such
a scenario (i.e., often much less because of delta compression, but in
your case of apparently-random media files, probably about 10MB).

I wasn't clear from your message: you indicated the changes you made,
but are you still having performance problems, or are you still waiting
to get data?

-Peff