From: Marco Costalba <hidden> Date: 2016-06-15 22:43:53
Hi all,
I have pushed a patch series to
git://git.kernel.org/pub/scm/qgit/qgit4.git
that changes the format of git log used to read data from a git repository.
Now instead of --pretty=raw a custom made --pretty=format is given,
this shrinks loaded data of 30% (17MB less on Linux tree) and gives a
good speed up when you are low on memory (especially on big repos)
Next step _would_ be to load log message body on demand (another 50%
reduction) but this has two drawbacks:
(1) Text search/filter on log message would be broken
(2) Slower to browse through revisions because for each revision an
additional git-rev-list /git-log command should be executed to read
the body
The second point is worsted by the fact that it is not possible to
keep a command running and "open" like as example git-diff-tree
--stdin and feed with additional revision's sha when needed. Avoiding
the burden to startup a new process each time to read a new log
message given an sha would let the answer much more quick especially
on lesser OS's
Indeed there is a git-rev-list --stdin option but with different
behaviour from git-diff-tree --stdin and not suitable for this.
Marco
From: Shawn O. Pearce <hidden> Date: 2016-06-15 22:43:54
Marco Costalba [off-list ref] wrote:
Now instead of --pretty=raw a custom made --pretty=format is given,
this shrinks loaded data of 30% (17MB less on Linux tree) and gives a
good speed up when you are low on memory (especially on big repos)
Next step _would_ be to load log message body on demand (another 50%
reduction) but this has two drawbacks:
(1) Text search/filter on log message would be broken
(2) Slower to browse through revisions because for each revision an
additional git-rev-list /git-log command should be executed to read
the body
The second point is worsted by the fact that it is not possible to
keep a command running and "open" like as example git-diff-tree
--stdin and feed with additional revision's sha when needed. Avoiding
the burden to startup a new process each time to read a new log
message given an sha would let the answer much more quick especially
on lesser OS's
Indeed there is a git-rev-list --stdin option but with different
behaviour from git-diff-tree --stdin and not suitable for this.
There was a proposed patch for git-cat-file that would let you run
it in a --stdin mode; the git-svn folks wanted this to speed up
fetching raw objects from the repository. That may help as you
could get commit bodies (in raw format - not reencoded format!)
quite efficiently.
Otherwise I think what you really want here is a libgit that you can
link into your process and that can efficiently inflate an object
on demand for you. Like the work Luiz was working on this past
summer for GSOC. Lots of downsides to that current tree though...
like die() kills the GUI...
--
Shawn.
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:43:54
Hi,
On Mon, 26 Nov 2007, Shawn O. Pearce wrote:
Marco Costalba [off-list ref] wrote:
quoted
Now instead of --pretty=raw a custom made --pretty=format is given,
this shrinks loaded data of 30% (17MB less on Linux tree) and gives a
good speed up when you are low on memory (especially on big repos)
Next step _would_ be to load log message body on demand (another 50%
reduction) but this has two drawbacks:
(1) Text search/filter on log message would be broken
(2) Slower to browse through revisions because for each revision an
additional git-rev-list /git-log command should be executed to read
the body
The second point is worsted by the fact that it is not possible to
keep a command running and "open" like as example git-diff-tree
--stdin and feed with additional revision's sha when needed. Avoiding
the burden to startup a new process each time to read a new log
message given an sha would let the answer much more quick especially
on lesser OS's
Indeed there is a git-rev-list --stdin option but with different
behaviour from git-diff-tree --stdin and not suitable for this.
There was a proposed patch for git-cat-file that would let you run
it in a --stdin mode; the git-svn folks wanted this to speed up
fetching raw objects from the repository. That may help as you
could get commit bodies (in raw format - not reencoded format!)
quite efficiently.
Otherwise I think what you really want here is a libgit that you can
link into your process and that can efficiently inflate an object
on demand for you. Like the work Luiz was working on this past
summer for GSOC. Lots of downsides to that current tree though...
like die() kills the GUI...
But then, die() calls die_routine, which you can override. And C++ has
this funny exception mechanism which just begs to be used here. The only
thing you need to add is a way to flush all singletons like the object
array.
Ciao,
Dscho
From: Marco Costalba <hidden> Date: 2016-06-15 22:43:54
On Nov 27, 2007 11:48 AM, Johannes Schindelin
[off-list ref] wrote:
quoted
quoted
Indeed there is a git-rev-list --stdin option but with different
behaviour from git-diff-tree --stdin and not suitable for this.
There was a proposed patch for git-cat-file that would let you run
it in a --stdin mode; the git-svn folks wanted this to speed up
fetching raw objects from the repository. That may help as you
could get commit bodies (in raw format - not reencoded format!)
quite efficiently.
That would be nice.
quoted
Otherwise I think what you really want here is a libgit that you can
link into your process and that can efficiently inflate an object
on demand for you.
I would think libgit is overkilling for this.
You probably would not use libgit to just add a single feature but to
change completely the interface with git because the required work is
heavy both on git side and qgit side (you probably would want to run
the libgit linked part in a separated thread to avoid GUI soft locks
during slow processing, now, because the executed git command is a
different process from qgit, the OS scheduler takes care of this 'for
free').
Marco
From: Jan Hudec <hidden> Date: 2016-06-15 22:43:54
On Tue, Nov 27, 2007 at 10:48:00 +0000, Johannes Schindelin wrote:
On Mon, 26 Nov 2007, Shawn O. Pearce wrote:
quoted
[...]
Otherwise I think what you really want here is a libgit that you can
link into your process and that can efficiently inflate an object
on demand for you. Like the work Luiz was working on this past
summer for GSOC. Lots of downsides to that current tree though...
like die() kills the GUI...
But then, die() calls die_routine, which you can override. And C++ has
this funny exception mechanism which just begs to be used here. The only
thing you need to add is a way to flush all singletons like the object
array.
Unfortunately, exceptions won't really work. Why? Because to use exceptions,
you need to have an exception-safe code. That is the code needs to free any
allocated resources when it's aborted by exception. And git code is not
exceptions safe. Given the lack of destructors in C, it means registering all
resource allocation in some kind of pool, so they can be freed en masse in
case of failure. Than you can also use longjmp for die (for C they really
behave the same).
--
Jan 'Bulb' Hudec [off-list ref]
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:43:54
Hi,
On Tue, 27 Nov 2007, Jan Hudec wrote:
On Tue, Nov 27, 2007 at 10:48:00 +0000, Johannes Schindelin wrote:
quoted
On Mon, 26 Nov 2007, Shawn O. Pearce wrote:
quoted
[...]
Otherwise I think what you really want here is a libgit that you can
link into your process and that can efficiently inflate an object
on demand for you. Like the work Luiz was working on this past
summer for GSOC. Lots of downsides to that current tree though...
like die() kills the GUI...
But then, die() calls die_routine, which you can override. And C++ has
this funny exception mechanism which just begs to be used here. The only
thing you need to add is a way to flush all singletons like the object
array.
Unfortunately, exceptions won't really work. Why? Because to use
exceptions, you need to have an exception-safe code. That is the code
needs to free any allocated resources when it's aborted by exception.
And git code is not exceptions safe. Given the lack of destructors in C,
it means registering all resource allocation in some kind of pool, so
they can be freed en masse in case of failure. Than you can also use
longjmp for die (for C they really behave the same).
Sorry, I just assumed that you can read my mind (or alternatively remember
what I suggested a few months ago, namely to "override" xmalloc(),
xcalloc(), xrealloc() and xfree() (probably you need to create the
latter)).
Ciao,
Dscho
On Tue, Nov 27, 2007 at 10:48:00 +0000, Johannes Schindelin wrote:
quoted
On Mon, 26 Nov 2007, Shawn O. Pearce wrote:
quoted
[...]
Otherwise I think what you really want here is a libgit that you can
link into your process and that can efficiently inflate an object
on demand for you. Like the work Luiz was working on this past
summer for GSOC. Lots of downsides to that current tree though...
like die() kills the GUI...
But then, die() calls die_routine, which you can override. And C++
has
quoted
this funny exception mechanism which just begs to be used here. The
only
quoted
thing you need to add is a way to flush all singletons like the object
array.
Unfortunately, exceptions won't really work. Why? Because to use
exceptions, you need to have an exception-safe code. That is the code
needs to free any allocated resources when it's aborted by exception.
And git code is not exceptions safe. Given the lack of destructors in C,
it means registering all resource allocation in some kind of pool, so
they can be freed en masse in case of failure. Than you can also use
longjmp for die (for C they really behave the same).
Sorry, I just assumed that you can read my mind (or alternatively remember
what I suggested a few months ago, namely to "override" xmalloc(),
xcalloc(), xrealloc() and xfree() (probably you need to create the
latter)).
That sounds like the easiest (but not necessarily easy) direction towards
the goal. Thread-local or global (I don't think git is currently reentrant
anyway) would do. Also filehanles would have to be taken care of and
everything checked for using malloc, calloc, strdup and other libc
functions directly.
Than die could longjmp out to a specified buffer and could be safely
overriden to throw exception for C++ apps.
--
- Jan Hudec [off-list ref]