From: Martin Langhoff <hidden> Date: 2016-06-15 22:42:17
On 2/1/06, Ray Lehtiniemi [off-list ref] wrote:
by various VAR companies. the tree in question has ~20,000 files
totalling nearly 1.4 GB
...
reiserfs$ time git update-index --refresh
If you have such a tree, your workflow _must_ be such that you know
exactly what files you have changed. Asking any tool to go out and
"find which of my 20K files has changed" is doable, but it's just
magic that it works on recent linuxes.
for comparison, one of our sandboxes is sitting on an NTFS file system,
accessed via SMB:
you have the samba stack, network, SMB/CIFS stack and NTFS itself in
the middle. Replace the ethernet with carrier pigeons for a more
complete picture ;-)
Perhaps a local git/cygwin on NTFS would be more reasonable to benchmark?
cheers,
martin
If you have such a tree, your workflow _must_ be such that you know
exactly what files you have changed. Asking any tool to go out and
"find which of my 20K files has changed" is doable, but it's just
magic that it works on recent linuxes.
It's not magic, and it's not all that recent. Linux FS ops have always
been pretty good, and the dentry cache was introduced in 2.0.x, I think,
so you'd be hard-pressed to find a Linux system that doesn't have it.
Now, I bet Linux will be better (often by a factor of 2-3) than most other
systems, but that still doesn't mean that 20k files is totally
unreasonable on other setups.
I suspect cygwin is worse than most because (a) the NT VFS layer is
piss-poor and you need a kernel service to get good performance and (b)
cygwin probably adds its own overhead for handling symlinks, so the
"lstat()" call is probably even more expensive.
Now, the networked filesystems are a potential problem for everybody.
Linus
$ time git update-index --refresh
real 0m21.500s
user 0m0.358s
sys 0m1.406s
WinNT, NTFS, 13k files, hot cache.
That's 25% less files than the Linux kernel, and I can do that operation
in 0m0.062s (0.012s user, 0.048s system).
So WinNT/cygwin is about 2.5 _orders_of_maginitude_ slower here, or 340
times slower.
Now, I'm tempted to say that NT is a piece of sh*t, but the fact is, your
CPU-times seem to indicate that most of it is IO (and the "real" cost is
just 1.7 seconds, much of which is system time, which in turn itself is
probably due to the IO costs too - so even that isn't comparable with
the ).
Which may mean that you simply don't have enough memory to cache the whole
thing. Which may be NT sucking, of course ("we don't like to use more than
10% of memory for caches"), but it might also be a tunable (which is sucky
in itself, of course), but finally, it might just be that you just don't
have a ton of memory. I've got 2GB in my machines, although 1GB is plenty
to cache the kernel.
Linus
From: "H. Peter Anvin" <hpa@zytor.com> Date: 2016-06-15 22:42:18
Linus Torvalds wrote:
It's not magic, and it's not all that recent. Linux FS ops have always
been pretty good, and the dentry cache was introduced in 2.0.x, I think,
so you'd be hard-pressed to find a Linux system that doesn't have it.
2.1.14, I seem to remember -- it was definitely 2.1.1x-ish. I mostly
recall because autofs didn't just break horribly, it took adding several
dcache hooks to make it work again :)
-hpa
From: Alex Riesen <hidden> Date: 2016-06-15 22:42:18
On 2/1/06, Linus Torvalds [off-list ref] wrote:
quoted
$ time git update-index --refresh
real 0m21.500s
user 0m0.358s
sys 0m1.406s
WinNT, NTFS, 13k files, hot cache.
That's 25% less files than the Linux kernel, and I can do that operation
in 0m0.062s (0.012s user, 0.048s system).
correction. It's 18k files, which is almost the same as 2.6.13-rc6. But these
files got *very* long names (the project poisoned by classical C++ education
and breaks windows' 255 chars limit on filename length from time to time).
Refresh index in 2.6.13 is actualy consistantly faster:
$ cd src/linux-2.6.13-rc6
$ time git update-index --refresh
real 0m1.344s
user 0m0.358s
sys 0m0.984s
So WinNT/cygwin is about 2.5 _orders_of_maginitude_ slower here, or 340
times slower.
Now, I'm tempted to say that NT is a piece of sh*t, but the fact is, your
CPU-times seem to indicate that most of it is IO (and the "real" cost is
just 1.7 seconds, much of which is system time, which in turn itself is
probably due to the IO costs too - so even that isn't comparable with
the ).
Which may mean that you simply don't have enough memory to cache the whole
thing. Which may be NT sucking, of course ("we don't like to use more than
10% of memory for caches"), but it might also be a tunable (which is sucky
in itself, of course), but finally, it might just be that you just don't
have a ton of memory. I've got 2GB in my machines, although 1GB is plenty
to cache the kernel.
I have 2Gb, the "System Cache" is around 1.5Gb, and this is PIV 3.2GHz.
There seem to be no tunables for any kind of system stuff
(savin' on support costs, do they?).
You'd be very hardpressed not to say that windows is a piece of sh*t.
The "benchmark: several times in a row:
$ time git update-index --refresh
real 0m1.766s
user 0m0.498s
sys 0m1.203s
$ time git update-index --refresh
real 0m1.766s
user 0m0.358s
sys 0m1.390s
$ time git update-index --refresh
real 0m1.781s
user 0m0.420s
sys 0m1.311s
$ time git update-index --refresh
real 0m1.875s
user 0m0.374s
sys 0m1.343s
$ time git update-index --refresh
real 0m1.766s
user 0m0.326s
sys 0m1.375s
It is always almost the same time. I don't think it's IO, looks more like
cache accesses. It is just that bad in this cygwin+win2k combination.
Besides, I don't trust "time <command>" on windows much: it returned
sys time 0 for git-update-index in a directory which was read before.
Yes, there was disk activity, I can hear it real good with that barrakuda.