With the help of Robin Rosenberg I've been able to make jgit's log
operation run (on average) within a few milliseconds of core Git.
Walking the 50,000 most recent commits from the Mozilla trunk[1]:
$ time git rev-list --max-count=50000 HEAD >/dev/null
core Git: 1.882s (average)
jgit: 1.932s (average)
(times are with hot cache and from repeated executions)
I think that is actually pretty good given that jgit is written
in Java using a fairly object-oriented design and has to deal with
some of the limitations of the language.
One of the biggest annoyances has been the fact that although Java
1.4 offers a way to mmap a file into the process, the overhead to
access that data seems to be far higher than just reading the file
content into a very large byte array, especially if we are going
to access that file content multiple times. So jgit performs worse
than core Git early on while it copies everything from the OS buffer
cache into the Java process, but then performs reasonably well once
the internal cache is hot. On the other hand using the mmap call
reduces early latency but hurts the access times so much that we're
talking closer to 3s average read times for the same log operation.
Anyway, jgit is now hopefully fast enough that we can start to build
some real functionality on top of it, and not need to wait several
minutes for answers from those features while debugging them. :)
**1** This is the pack file from Jon Smirl's import attempt.
--
I tried to access that with git 1.4.4.1 from Debian but
% git clone http://www.spearce.org/projects/scm/egit.git
hangs, the first time after "walk e339766abc2b919e7bb396cae22ddef065821381",
the second time after "walk 9eec90ec5da239e063eaff6305d77294dc03396e"
which is the "walk" line just before it.
There's also the following error shortly after the start:
error: File bc01ab9e5fcd26918d7a334207183fa57ff1ce50 (http://www.spearce.org/projects/scm/egit.git/objects/75/1c8f2e504c40d1c41ebbd87d8f8968529e9c30) corrupt
Jürgen
--
Jürgen Stuber [off-list ref]
http://www.jstuber.net/
gnupg key fingerprint = 2767 CA3C 5680 58BA 9A91 23D9 BED6 9A7A AF9E 68B4
From: Jakub Narebski <hidden> Date: 2016-08-11 19:25:10
Robin Rosenberg wrote:
söndag 03 december 2006 05:59 skrev Shawn Pearce:
quoted
With the help of Robin Rosenberg I've been able to make jgit's log
operation run (on average) within a few milliseconds of core Git.
Walking the 50,000 most recent commits from the Mozilla trunk[1]:
$ time git rev-list --max-count=50000 HEAD >/dev/null
core Git: 1.882s (average)
jgit: 1.932s (average)
(times are with hot cache and from repeated executions)
Nice indeed. That was a ten-fold improvement for getting my full history.
So, just go on to the next case. I added filtering on filenames (yes,
CVS-induced brain damage, I should track the content. next version. filenames
are so much handier to work with).
Git uses <path> as _revision limiter_, not as output filter. Shouldn't
jgit do the same?
P.S. What is the status of --follow option?
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
Right. My web server doesn't send back 404 messages when an object
isn't found (but is packed). Thus the error...
I've setup a project on repo.or.cz:
http://repo.or.cz/w/egit.git
--
Thanks. That particular bug has been of some discussion on
#git before. I'm planning on making it a configuration flag in
.git/config for the user to decide how much pain they want: mmap
(with all its huge downsides) or read into byte[] (with all the
memory footprint that requires).
--
So, just go on to the next case. I added filtering on filenames (yes,
CVS-induced brain damage, I should track the content. next version. filenames
are so much handier to work with). That gives me 4.5s to retrieve a filtered
history (from 10800 commits).Half of the time is spent in re-sorting tree
entries. Is that really necessary?
Yea, I was looking at that code while doing the other performance
improvements and thought it might start to become a bottleneck. I
guess I was right.
What is happening here is jgit wants to store the items in the tree
in name ordering, but Git stores the items in the tree sorted such
that subtrees sort with a '/' on the end of their name. This is a
different ordering...
The reason I'm resorting them is so we can find an entry without
knowing what its type is first. Looks like that's going to have
to change somehow.
Most of java's slowness comes from the programmers using it. (Lutz Prechelt.
Technical opinion: comparing Java vs. C/C++ efficiency differences to
interpersonal differences. ACM, Vol 42,#10, 1999)
Yes, that was clearly the case here with jgit! :-)
_This_ programmer made jgit slow. Learned from the mistake, and
made it faster.
quoted
One of the biggest annoyances has been the fact that although Java
1.4 offers a way to mmap a file into the process, the overhead to
access that data seems to be far higher than just reading the file
content into a very large byte array, especially if we are going
to access that file content multiple times. So jgit performs worse
than core Git early on while it copies everything from the OS buffer
cache into the Java process, but then performs reasonably well once
the internal cache is hot. On the other hand using the mmap call
reduces early latency but hurts the access times so much that we're
talking closer to 3s average read times for the same log operation.
Have you tried that with difference JVM's?
No, I'm on Mac OS X so I don't have a huge JVM selection (that I
know of). And I haven't tried jgit or egit on any other system yet.
--
söndag 03 december 2006 15:19 skrev Jakub Narebski:
quoted
Robin Rosenberg wrote:
quoted
CVS-induced brain damage, I should track the content. next version.
filenames are so much handier to work with).
Git uses <path> as _revision limiter_, not as output filter. Shouldn't
jgit do the same?
It's egit, i.e. the eclipse plugin I'm referring to so it's a user interface
thing and it uses the path name.
Jakub's point is that "git log -- a" lists only the revisions
which affect path 'a'. Once those revisions have been selected
then output begins. The 'a' gets reapplied as an output filter to
only show changes relevant to file 'a', but its very much a revision
filter during the revision listing process.
--
From: Jakub Narebski <hidden> Date: 2016-08-11 20:06:26
Linus Torvalds wrote:
Also, I have to say, one of the reasons I'm interested in your project is
that I've never done any Java programming, because quite frankly, I've
never had any reason what-so-ever to do so. But if there is some simple
setup, and you have jgit exposed somewhere as a git archive, I'd love to
take a look, if only to finally learn more about Java.
I tried to access that with git 1.4.4.1 from Debian but
% git clone http://www.spearce.org/projects/scm/egit.git
hangs, the first time after "walk
e339766abc2b919e7bb396cae22ddef065821381", the second time after "walk
9eec90ec5da239e063eaff6305d77294dc03396e" which is the "walk" line just
before it.
Works fine here. (git 1.4.4.gf05d).
now it works fine for me, too.
Tack
Jürgen
--
Jürgen Stuber [off-list ref]
http://www.jstuber.net/
gnupg key fingerprint = 2767 CA3C 5680 58BA 9A91 23D9 BED6 9A7A AF9E 68B4
I tried to access that with git 1.4.4.1 from Debian but
% git clone http://www.spearce.org/projects/scm/egit.git
hangs, the first time after "walk
e339766abc2b919e7bb396cae22ddef065821381", the second time after "walk
9eec90ec5da239e063eaff6305d77294dc03396e" which is the "walk" line just
before it.
Unfortunately, messages about corrupt objects are "normal" with clone over
http. I'm not sure it has to be that way though. Run git-fsck-objects to make
sure there are no errors. The hangs aren't normal.
-- robin
With the help of Robin Rosenberg I've been able to make jgit's log
operation run (on average) within a few milliseconds of core Git.
Very good. Are we any closer to actually having an eclipse plugin then?
Not that I've ever actually used eclipse, but maybe I should try it, just
to see what those strange user-land people actually do. I'll be a
veritable Jane Goodall..
Walking the 50,000 most recent commits from the Mozilla trunk[1]:
$ time git rev-list --max-count=50000 HEAD >/dev/null
core Git: 1.882s (average)
jgit: 1.932s (average)
(times are with hot cache and from repeated executions)
Now, the _interesting_ case in many ways is not "--max-count", but the
revision limiter. It _should_ be equally fast, but if you've done
something wrong, it won't be.
IOW, try to find a point far enough back in time to get about the same
number of commits, and then do
time git rev-list <thatpoint>..HEAD >/dev/null
because one of the things you want to handle is ranges, more so than
simple counts. And that is not only the much more common case, it also
triggers a few cases that you probably didn't trigger with the regular
"list the first 50 thousand commits" case.
One of the biggest annoyances has been the fact that although Java
1.4 offers a way to mmap a file into the process, the overhead to
access that data seems to be far higher than just reading the file
content into a very large byte array, especially if we are going
to access that file content multiple times.
That must suck for big packed repositories. What JVM and other environment
are you using?
Also, I have to say, one of the reasons I'm interested in your project is
that I've never done any Java programming, because quite frankly, I've
never had any reason what-so-ever to do so. But if there is some simple
setup, and you have jgit exposed somewhere as a git archive, I'd love to
take a look, if only to finally learn more about Java.
I tried to access that with git 1.4.4.1 from Debian but
% git clone http://www.spearce.org/projects/scm/egit.git
hangs, the first time after "walk
e339766abc2b919e7bb396cae22ddef065821381", the second time after "walk
9eec90ec5da239e063eaff6305d77294dc03396e" which is the "walk" line just
before it.
Unfortunately, messages about corrupt objects are "normal" with clone over
http. I'm not sure it has to be that way though. Run git-fsck-objects to make
sure there are no errors. The hangs aren't normal.
I got:
$ git clone http://www.spearce.org/projects/scm/egit.git
[...]
got 73ed47b2bb1fa5978f7368775979e5c85d354c5a
error: File 2332eacf114debb7a27d138811197f06eb262551
(http://www.spearce.org/projects/scm/egit.git/objects/75/1c8f2e504c40d1c41ebbd87d8f8968529e9c30) corrupt
Getting pack list for http://www.spearce.org/projects/scm/egit.git/
got afefbe09bacc08adb75fb46200a973001c6b02de
[...]
walk c1f287cb19b9910af19756cf29c08b1fda75da8c
Some loose object were found to be corrupt, but they might be just
a false '404 Not Found' error message sent with incorrect HTTP
status code. Suggest running git fsck-objects.
got eab86de8ac23e2e77878835007724146fdd83796
$ git fsck-objects --unreachable --full --strict ;# returns no errors
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
Very good. Are we any closer to actually having an eclipse plugin then?
We have some parts in place. But nothing that's user-ready.
quoted
Walking the 50,000 most recent commits from the Mozilla trunk[1]:
Now, the _interesting_ case in many ways is not "--max-count", but the
revision limiter. It _should_ be equally fast, but if you've done
something wrong, it won't be.
We haven't implemented a rev-list equivalent yet. That's a major
feature which is missing. The --max-count test was simple to put
together. I really need to start thinking about replicating some
of the features of the revision functions in core Git.
quoted
One of the biggest annoyances has been the fact that although Java
1.4 offers a way to mmap a file into the process, the overhead to
access that data seems to be far higher than just reading the file
content into a very large byte array, especially if we are going
to access that file content multiple times.
That must suck for big packed repositories. What JVM and other environment
are you using?
Mac OS 10.4.8 / Java 1.4.2. It appears as though Sun isn't going
to fix the mmap performance problems as they can't do it "securely".
So its likely to be an issue anywhere jgit gets used...
Also, I have to say, one of the reasons I'm interested in your project is
that I've never done any Java programming, because quite frankly, I've
never had any reason what-so-ever to do so. But if there is some simple
setup, and you have jgit exposed somewhere as a git archive, I'd love to
take a look, if only to finally learn more about Java.
Of course its in Git. :-)
There's no webpage here, but you can clone it:
http://www.spearce.org/projects/scm/egit.git
The code you would recognize most is in org.spearce.jgit/.../lib.
E.g. BinaryDelta.java came from patch-delta.c in core Git.
Repository has some of the functionality of sha1_file.c, the pack
reading code is in PackFile.java.
--
From: Robin Rosenberg <hidden> Date: 2016-08-11 20:40:26
söndag 03 december 2006 05:59 skrev Shawn Pearce:
With the help of Robin Rosenberg I've been able to make jgit's log
operation run (on average) within a few milliseconds of core Git.
Walking the 50,000 most recent commits from the Mozilla trunk[1]:
$ time git rev-list --max-count=50000 HEAD >/dev/null
core Git: 1.882s (average)
jgit: 1.932s (average)
(times are with hot cache and from repeated executions)
Nice indeed. That was a ten-fold improvement for getting my full history.
So, just go on to the next case. I added filtering on filenames (yes,
CVS-induced brain damage, I should track the content. next version. filenames
are so much handier to work with). That gives me 4.5s to retrieve a filtered
history (from 10800 commits).Half of the time is spent in re-sorting tree
entries. Is that really necessary?
I think that is actually pretty good given that jgit is written
in Java using a fairly object-oriented design and has to deal with
some of the limitations of the language.
Most of java's slowness comes from the programmers using it. (Lutz Prechelt.
Technical opinion: comparing Java vs. C/C++ efficiency differences to
interpersonal differences. ACM, Vol 42,#10, 1999)
One of the biggest annoyances has been the fact that although Java
1.4 offers a way to mmap a file into the process, the overhead to
access that data seems to be far higher than just reading the file
content into a very large byte array, especially if we are going
to access that file content multiple times. So jgit performs worse
than core Git early on while it copies everything from the OS buffer
cache into the Java process, but then performs reasonably well once
the internal cache is hot. On the other hand using the mmap call
reduces early latency but hurts the access times so much that we're
talking closer to 3s average read times for the same log operation.