Hi All,
I was thinking about possible ideas for my little pet project and I
had and idea for way to tack on notes to a commit, or any object
really. I know that the idea has been flying around for a long time
but there has never been any implementation or a concept that people
liked enough to use (unless I have missed something).
Here is my idea.
.git/refs/notes contains a tree-id (assuming that using a tree-id
will not cause any problems, otherwise a commit object can be used.
it does not *need* a history, but it *could* have one).
That tree has a structure similar to the layout of .git/objects, where
it is 2 letter subdirectories for the notes objects.
Given a git object (commit, tree, blob, tag), use its sha as the
path/filename in this tree.
If I have a commit 1234567890123456789012345678901234567890 then
the notes tree will have a file
12/34567890123456789012345678901234567890
That file has a list of sha1s (one per line). These shas are object
IDs for blobs that have the notes or whatever that you want attached
to the item.
I think you get the idea. When looking up an item, it should be
fairly easy to have the notes tree and subtrees available for doing
lookups. And as far as I know stuff under .git/refs can be
pushed/pulled even if its not under heads or remotes or tags using
already existing machinery. I am not sure, but I think that would
satisfy gc operations as well. Also, these trees and blobs never have
to be put in the working directory.
Does this sound like something that is workable? I thought it might
appeal since it uses only features that are already present.
This could be extended so that you have different sets of notes under
.git/refs/notes/<my note set> or whatever. So that you can have some
notes you keep private and some that you publish or whatever.
OK, hopefully this isn't a off the wall,
thats-what-you-get-for-being-up-at-2-AM idea.
Thanks,
Govind.
From: Jeff King <hidden> Date: 2016-06-15 22:45:47
On Tue, Dec 16, 2008 at 02:15:47AM -0600, Govind Salinas wrote:
I was thinking about possible ideas for my little pet project and I
had and idea for way to tack on notes to a commit, or any object
really. I know that the idea has been flying around for a long time
but there has never been any implementation or a concept that people
liked enough to use (unless I have missed something).
I think you look at the previous suggestions, because yours is very
similar. Which is good, I think, because the current status is that the
design is good, but nobody has gotten around to working on it yet. So
maybe you can fix that. :)
.git/refs/notes contains a tree-id (assuming that using a tree-id
will not cause any problems, otherwise a commit object can be used.
it does not *need* a history, but it *could* have one).
That is the same as the current proposal, except:
- the proposal is to use a commit, so your notes are version-controlled
- I have suggested supporting multiple note "bases" in the refs/notes
namespace. This would allow you to share some notes but not others
(e.g., if you had some automated notes related to a build/test
system, you might not want to mix those with your human-written
notes).
That tree has a structure similar to the layout of .git/objects, where
it is 2 letter subdirectories for the notes objects.
I don't think this has been suggested yet, but I'm not sure it is a good
idea. The usual reason for this split is that many filesystems handle
large directories badly; that isn't a problem here.
It does reduce the size of the resulting tree objects when a note is
modified (we make updates to two smaller trees instead of one big tree).
I don't know if this really matters all that much, since the trees
will end up deltified in a pack anyway.
And it does make the implementation slightly less simple, since we have
to deal with two levels of trees.
Given a git object (commit, tree, blob, tag), use its sha as the
path/filename in this tree.
If I have a commit 1234567890123456789012345678901234567890 then
the notes tree will have a file
12/34567890123456789012345678901234567890
That file has a list of sha1s (one per line). These shas are object
IDs for blobs that have the notes or whatever that you want attached
to the item.
This is slightly different than the current proposal. You are proposing
that each item have a "list of notes". My thinking was to have "named
notes" using a tree for each entry full of blobs. So you could look up
the "foo" note for a given commit, but that note would be a single
scalar (which could, of course, be interpreted according to its name).
I think you get the idea. When looking up an item, it should be
fairly easy to have the notes tree and subtrees available for doing
lookups. And as far as I know stuff under .git/refs can be
It is easy, but it's slow because we have to do a linear search in the
(potentially huge) notes tree. And that's what held up the initial
implementation. I did a proof-of-concept a month or so ago that
pre-seeded an in-memory hash using the tree contents and got pretty
reasonable performance.
pushed/pulled even if its not under heads or remotes or tags using
already existing machinery. I am not sure, but I think that would
satisfy gc operations as well. Also, these trees and blobs never have
to be put in the working directory.
Right, though I think one of the benefits of this approach is that you
_could_ do a checkout on the notes tree if you wanted to do very
flexible editing.
Does this sound like something that is workable? I thought it might
appeal since it uses only features that are already present.
Yes, it sounds workable, though if you diverge from what has already
been discussed, I think you should make an argument about why your
approach is better.
This could be extended so that you have different sets of notes under
.git/refs/notes/<my note set> or whatever. So that you can have some
notes you keep private and some that you publish or whatever.
From: Jeff King <hidden> Date: 2016-06-15 22:45:47
On Tue, Dec 16, 2008 at 03:51:08AM -0500, Jeff King wrote:
I think you look at the previous suggestions, because yours is very
Sorry, there is a typo there. I meant "I think you _should_ look at the
previous suggestions." Not saying in broken English that you already
have looked at them.
-Peff
On Tue, Dec 16, 2008 at 2:51 AM, Jeff King [off-list ref] wrote:
On Tue, Dec 16, 2008 at 02:15:47AM -0600, Govind Salinas wrote:
quoted
I was thinking about possible ideas for my little pet project and I
had and idea for way to tack on notes to a commit, or any object
really. I know that the idea has been flying around for a long time
but there has never been any implementation or a concept that people
liked enough to use (unless I have missed something).
I think you look at the previous suggestions, because yours is very
similar. Which is good, I think, because the current status is that the
design is good, but nobody has gotten around to working on it yet. So
maybe you can fix that. :)
I was thinking I would do my first implementation in pyrite and if I find
that it works well I will port it.
quoted
.git/refs/notes contains a tree-id (assuming that using a tree-id
will not cause any problems, otherwise a commit object can be used.
it does not *need* a history, but it *could* have one).
That is the same as the current proposal, except:
- the proposal is to use a commit, so your notes are version-controlled
- I have suggested supporting multiple note "bases" in the refs/notes
namespace. This would allow you to share some notes but not others
(e.g., if you had some automated notes related to a build/test
system, you might not want to mix those with your human-written
notes).
quoted
That tree has a structure similar to the layout of .git/objects, where
it is 2 letter subdirectories for the notes objects.
I don't think this has been suggested yet, but I'm not sure it is a good
idea. The usual reason for this split is that many filesystems handle
large directories badly; that isn't a problem here.
I just read the proposal from Johannes, he seems to want to use a
similar layout. However, I would like to modify my proposal slightly
to make it work better when a gc is run. I would modify the tree to
look like this...
let 1234567890123456789012345678901234567890 be the
id of the item that is annotated.
let abcdef7890123456789012345678901234567890 be the
id of the note to be attached
root/
12/
34567890123456789012345678901234567890/
abcdef7890123456789012345678901234567890
This way all the notes are attached to a tree, so that gc won't
think they are unreferenced objects.
It does reduce the size of the resulting tree objects when a note is
modified (we make updates to two smaller trees instead of one big tree).
I don't know if this really matters all that much, since the trees
will end up deltified in a pack anyway.
And it does make the implementation slightly less simple, since we have
to deal with two levels of trees.
quoted
Given a git object (commit, tree, blob, tag), use its sha as the
path/filename in this tree.
If I have a commit 1234567890123456789012345678901234567890 then
the notes tree will have a file
12/34567890123456789012345678901234567890
That file has a list of sha1s (one per line). These shas are object
IDs for blobs that have the notes or whatever that you want attached
to the item.
This is slightly different than the current proposal. You are proposing
that each item have a "list of notes". My thinking was to have "named
notes" using a tree for each entry full of blobs. So you could look up
the "foo" note for a given commit, but that note would be a single
scalar (which could, of course, be interpreted according to its name).
quoted
I think you get the idea. When looking up an item, it should be
fairly easy to have the notes tree and subtrees available for doing
lookups. And as far as I know stuff under .git/refs can be
It is easy, but it's slow because we have to do a linear search in the
(potentially huge) notes tree. And that's what held up the initial
implementation. I did a proof-of-concept a month or so ago that
pre-seeded an in-memory hash using the tree contents and got pretty
reasonable performance.
Perhaps I am missing something, how is it a linear search?. Since we
are keying off of the sha of the annotated object, using a hashtable for
a cache should be a fairly quick binary search. If you just wanted to
use the tree objects, that should work almost as well since the first tree
will split them up nicely for you.
Also, how large do you expect the list to be under reasonable
circumstances.
quoted
pushed/pulled even if its not under heads or remotes or tags using
already existing machinery. I am not sure, but I think that would
satisfy gc operations as well. Also, these trees and blobs never have
to be put in the working directory.
Right, though I think one of the benefits of this approach is that you
_could_ do a checkout on the notes tree if you wanted to do very
flexible editing.
Sure, why not.
quoted
Does this sound like something that is workable? I thought it might
appeal since it uses only features that are already present.
Yes, it sounds workable, though if you diverge from what has already
been discussed, I think you should make an argument about why your
approach is better.
Well after reading Johannes proposal, I find it to be surprisingly similar
since I had not seen it before. However I think mine is a win in a few ways.
One, it allows multiple notes per object. Two, it plays well with gc. From
what I could follow, the files in his layout just have a ref to the note object
but gc would be required to know about this feature and not remove those
note blobs. Third it allows multiple sets of notes. Although it seems that
at least 1 and 3 have been discussed at some length.
quoted
This could be extended so that you have different sets of notes under
.git/refs/notes/<my note set> or whatever. So that you can have some
notes you keep private and some that you publish or whatever.
Oops, I should have read your whole mail. Yes, that is a good idea. :)
For reference, here are the previous discussions that I think are
relevant:
Thanks for the pointers, a couple quick thoughts...
On rebasing, I have a couple thoughts. 1) I think it only really makes
sense to make a public annotation to a commit that is public, and
once a commit is public it should not be rebased. 2) We could also
annotate the commit's tree instead of the commit. That would make
it somewhat resistant to rebases, cherry-picks and amends. And
once a tree has changed, the notes are probably less reliable
although the user should be able to force a note or notes to be
carried along.
On naming. I strongly support a ref/notes/sha1/sha1 approach. If
having a type to the note is important, then perhaps the first line of
a note could be considered a type or a set of "tags". This way you
have both naming/typing and one lookup per sha. The only
drawback is that you have to open the blob to see the type. A hybrid
approach that uses refs/notes/acked/sha/sha which is one lookup if
you know the type and the sha of the annotated object before hand
might be worth considering. This would be similar to the public or
private notes that i mentioned before.
I guess I must be missing something. I have seen several references
to this not being a binary search several times in the links that you have
here. But I fail to see why a binary search cannot be done. That said, I
would still think that the existing hash table would be the way to go.
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:45:47
Hi,
On Tue, 16 Dec 2008, Govind Salinas wrote:
I was thinking I would do my first implementation in pyrite and if I
find that it works well I will port it.
Given that there are a lot of building blocks in C already, I think it
would be a waste of your time.
I just read the proposal from Johannes, he seems to want to use a
similar layout. However, I would like to modify my proposal slightly to
make it work better when a gc is run. I would modify the tree to look
like this...
let 1234567890123456789012345678901234567890 be the
id of the item that is annotated.
let abcdef7890123456789012345678901234567890 be the
id of the note to be attached
root/
12/
34567890123456789012345678901234567890/
abcdef7890123456789012345678901234567890
This way all the notes are attached to a tree, so that gc won't
think they are unreferenced objects.
In my proposal back then, your root/12/345... would be a blob, in Peff's,
it would be a tree, and in both cases the blobs/trees would be referenced
by refs/notes, so git gc would not kill them either.
The bigger issue is that commit objects can be gc'ed, and then their notes
should be gc'ed, too.
And of course, there is the rebase issue (which I completely missed; I
will read the mail Jeff referenced tomorrow).
Speaking about the blobs vs trees issue, I think it is no issue, as Peff
and me already discussed: the notes could check if it is a tree or a
blob, and handle both easily.
Peff wrote:
quoted
It is easy, but it's slow because we have to do a linear search in the
(potentially huge) notes tree. And that's what held up the initial
implementation. I did a proof-of-concept a month or so ago that
pre-seeded an in-memory hash using the tree contents and got pretty
reasonable performance.
Perhaps I am missing something, how is it a linear search?
Yes, you are missing what I wrote in the original thread: tree objects
must be read in a forward direction, one by one.
IIRC back then, Junio and/or Linus suggested that you could backward
search with heuristics finding the beginning of a tree entry, and thus you
could kind of bisect the tree to search for a specific tree entry (since
tree objects have the contents sorted), but I presented a case where this
breaks down at the GitTogether:
Tree entries consist of a mode (as 6-byte ASCII representing the octal
value), then SPC, then a NUL-terminated path, and then a 20-byte SHA-1.
(Just hexdump the output of "git cat-file tree HEAD:" in any repository to
see it).
The only heuristic you could apply to find your way backward (or forward)
to find the beginning of a tree entry would be to find the NUL character
and verify that exactly 20 bytes after it, either the tree object ends, or
there is a valid octal number followed by a SPC.
The only thing you would need for this heuristic to break down is a SHA-1
which contains a \x00 (which is then mistaken for a string termination),
and part of a filename that could be mistaken to be an octal number.
Take for example the SHA-1 of git-gui in v1.6.0.5, which has a NUL its
14th byte, and just assume that the next tree entry has mode 100644
and name "4040000 Some financial record.txt".
Then, the NUL could be mistaken for the end of the previous tree entry,
and "040000 " as the mode for the next one, whose name would be assumed to
be "Some financial record.txt".
Granted, if you find two NULs in 21 bytes, one of them must be the
termination of the path, but which one?
Worse, even if you would find a method (complicated, and therefore
necessarily fragile) to find the boundary of the tree entries reliably,
you would _still_ have a linear time unpacking the darned tree object in
the first place.
So you cannot do that for every commit you encounter and expect not to die
of boredom in the process.
Peff's very cute idea was to decouple that process from the per-commit
procedure, and basically make it a one-time cost (per Git call, and only
when notes were asked for).
It will still be linear in the number of notes, but it would then be in a
hashmap, with an expected linear cost per commit.
Also, how large do you expect the list to be under reasonable
circumstances.
We did not intend Git to be used as a backup tool, did we?
One of the _worst_ design decisions is to limit yourself by expectations.
On rebasing, I have a couple thoughts.
1) I think it only really makes sense to make a public annotation to a
commit that is public, and once a commit is public it should not be
rebased.
Again, do not limit your design by your expectations. People already talk
about having cover letters for their patch series as notes, and Pasky
seems to discuss tracking explicit renames with notes when he does not
play Go, instead of maintaining repo.or.cz and git.or.cz.
2) We could also annotate the commit's tree instead of the commit.
That would make it somewhat resistant to rebases, cherry-picks and
amends.
To the contrary. When I rebase, the tree _does_ change, otherwise I would
have rebased onto something that had the same original tree as my
rebase-base to begin with, which would make the rebase rather pointless.
On naming. I strongly support a ref/notes/sha1/sha1 approach.
I think you meant refs/notes:<first byte in hex>/<rest of bytes>/<some
arbitrary SHA-1>?
I am rather supporting refs/nots:<first byte in hex>/<rest of bytes> being
either a blob, or a tree containing human readable tags, such as "bugfix"
or "review" or some such.
If having a type to the note is important, then perhaps the first line
of a note could be considered a type or a set of "tags".
That would be horrible! Just to know if you need to unpack the blob,
you'd have to unpack it!
Hth,
Dscho
Oh, I misinterpreted that label... of course you can track rebases in
notes, but some issue that we did not look into yet (I think) is the issue
that you can cherry-pick and rebase commits and lose notes in the process.
It seems that the notes idea is not that unintrusive as I thought...
Ciao,
Dscho
Oh, I misinterpreted that label... of course you can track rebases in
notes, but some issue that we did not look into yet (I think) is the
issue that you can cherry-pick and rebase commits and lose notes in the
process.
It seems that the notes idea is not that unintrusive as I thought...
So we have two issues here:
1. Using notes to annotate the rebase/cherry-pick action itself.
2. Preserving (or at least giving the user the option of preserving) notes
across a rebase/cherry-pick.
I think issue #1 has already been discussed, and is largely resolved: People
can do this if they want to; it probably only makes sense when
rebasing/cherry-picking public branches; etc... AFAICS there are no
remaining problems here that needs an intrusive solution (see below for one
such unintrusive alternative).
Issue #2, however, is a little more involved. We can discuss the merits of
wanting to preserve notes across a rebase/cherry-pick itself; e.g. when it
makes sense to preserve notes, and when it doesn't make sense, but I think
this is orthogonal to the issue of HOW to preserve them, so instead of
focusing on WHY, I'll focus on HOW:
If notes are named according to the "refs/notes:<first byte in hex>/<rest of
bytes>/<referenced object SHA-1>" scheme (and AFAICS this is still being
discussed, so it's indeed a big IF), then rebase/cherry-pick of the
referenced object simply translates to a rename/copy of the corresponding
note (this is of course assuming that the note itself does not contain the
SHA-1 of the referenced object). This could probably be solved fairly
unintrusively in the current code, but there are (as always) complications:
- The user may want to amend the note after the rebase/cherry-pick (just as
(s)he may want to amend the commit message).
- In some cases it may even make sense to fold (parts of) the note _into_
the commit message.
- probably more reasons...
So what about the following proposal: Add hooks that are invoked by
rebase/cherry-pick with the <from-SHA1> and <to-SHA1> as arguments. A
typical hook script can then use this information to look up notes
referencing <from-SHA1> and update these to reference <to-SHA1> instead,
and in the process, prompt the user to do whatever changes (s)he wants to.
The hook scripts can do other things as well, e.g. implementing issue #1
above (adding notes for annotating the rebase/cherry-pick itself.)
Have fun! :)
...Johan
PS: What's the current status on git-sequencer? It's probably the best place
to invoke these hooks.
--
Johan Herland, [off-list ref]
www.herland.net
From: Jeff King <hidden> Date: 2016-06-15 22:45:47
On Tue, Dec 16, 2008 at 12:43:55PM -0600, Govind Salinas wrote:
I was thinking I would do my first implementation in pyrite and if I find
that it works well I will port it.
OK, though your performance will probably suck unless you dump the notes
tree into a local hash at the beginning of your program. And looking up
every commit's note during revision traversal is one of the intended
uses (e.g., decorating git-log output, or filtering commits based on a
particular note).
And as Dscho mentioned, most of what you need is already there in C.
You are welcome to implement whatever you want in pyrite, of course, but
there is a desire to have this accessible to the revision traversal
machinery. And that means if you want your version in pyrite to be
compatible with what ends up in git, the data structure design needs to
be suitable for both.
I just read the proposal from Johannes, he seems to want to use a
similar layout. However, I would like to modify my proposal slightly
to make it work better when a gc is run. I would modify the tree to
look like this...
let 1234567890123456789012345678901234567890 be the
id of the item that is annotated.
let abcdef7890123456789012345678901234567890 be the
id of the note to be attached
root/
12/
34567890123456789012345678901234567890/
abcdef7890123456789012345678901234567890
This way all the notes are attached to a tree, so that gc won't
think they are unreferenced objects.
But you have lost the ordering in your list, then, since they will not
be ordered by sha1 of the note contents. I don't know if you care. The
second sha1 is pointless, anyway, since nobody will know that number as
a reference; why not just name them monotonically starting at 1?
One of the things I don't like about having several notes is that it
introduces an extra level of indirection that every user has to pay for,
whether they want it or not. If a note can be a blob _or_ a tree, then
those who want to use blobs can reap the performance benefit. Those who
want multiple named notes in a hierarchy can pay the extra indirection
cost.
I haven't measured how big a cost that is (but bearing in mind that we
might want to do this lookup once per revision in a traversal, even one
extra object lookup can have an impact).
I'm also still not convinced the fan-out is worthwhile, but I can see
how it might be. It would be nice to see numbers for both.
Perhaps I am missing something, how is it a linear search?. Since we
I think Johannes explained in detail in another message, but it is a
linear search to look up directly in a tree object. Of course you can
build a hash or a sorted fixed-size list as an index.
Also, how large do you expect the list to be under reasonable
circumstances.
As many notes as there are commits is my goal (e.g., it is not hard to
imagine an automated process to add notes on build status). Ideally, we
could handle as many notes as there are objects; I see no reason not to
allow annotating arbitrary sha1's (I don't know if there is a use for
that, but the more scalable the implementation, the better).
On naming. I strongly support a ref/notes/sha1/sha1 approach. If
having a type to the note is important, then perhaps the first line of
a note could be considered a type or a set of "tags". This way you
I don't think we are talking about the same thing. What I mean by naming
is "here is a shorthand for referring to notes" that is not necessarily
coupled with the implementation. That is, I would like to do something
like:
git log --notes-filter="foo:bar == 1"
and have that "foo:bar" as a shorthand on each commit for:
refs/notes/foo:$COMMIT/bar
Without a left-hand side (e.g., "bar"), we get:
refs/notes/default:$COMMIT/bar
Or without a right-hand side (e.g., "foo:"), we get:
refs/notes/foo:$COMMIT
So you can group related notes in the same tree (which gives you fast
lookup if you are looking at multiple ones, since you only have to do
the tree lookup once), or you can keep notes in separate trees (which
means you can distribute some but not others).
I think your "list of notes" proposal on top of that would be for any
note resolution to provide a tree instead of a blob, with sequenced
elements. I.e., foo:bar might have multiple notes, like:
refs/notes/foo:$COMMIT/bar/1
refs/notes/foo:$COMMIT/bar/2
refs/notes/foo:$COMMIT/bar/3
drawback is that you have to open the blob to see the type. A hybrid
approach that uses refs/notes/acked/sha/sha which is one lookup if
Right, that is possible, but implementing only half of what I suggested
above. I think you should be flexible enough to have grouped notes for
fast lookup, or ungrouped notes for more flexibility.
-Peff
From: Jeff King <hidden> Date: 2016-06-15 22:45:47
On Wed, Dec 17, 2008 at 12:48:02AM +0100, Johannes Schindelin wrote:
quoted
Perhaps I am missing something, how is it a linear search?
Yes, you are missing what I wrote in the original thread: tree objects
must be read in a forward direction, one by one.
Thanks, that was a nice writeup of what was discussed at the
GitTogether.
Peff's very cute idea was to decouple that process from the per-commit
procedure, and basically make it a one-time cost (per Git call, and only
when notes were asked for).
To be fair, it was not my cute idea, but somebody else's (I think David
Reiss). I just coded it quickly because somebody was talking about
iPhones or something. :)
To the contrary. When I rebase, the tree _does_ change, otherwise I
would have rebased onto something that had the same original tree as
my rebase-base to begin with, which would make the rebase rather
pointless.
Another fun option would be to put notes on patch-ids. That would of
course be horrifically slow to look up, but would survive many
cherry-picks and rebases (but not all, of course).
I don't know if that is useful or not, but I don't see any reason why
the discussed implementation would forbid it (somebody would just have
to implement the lookups at a useful spot).
quoted
On naming. I strongly support a ref/notes/sha1/sha1 approach.
I think you meant refs/notes:<first byte in hex>/<rest of bytes>/<some
arbitrary SHA-1>?
You seem to be in favor of the fan-out. Out of curiosity, did you ever
do numbers on whether the fan-out is actually helpful?
I am rather supporting refs/nots:<first byte in hex>/<rest of bytes>
being either a blob, or a tree containing human readable tags, such as
"bugfix" or "review" or some such.
Yes, I am in favor of the "tree or blob" idea, too. I want this to not
just be about "I am a person writing a note" but "arbitrary data
attached to an object after it has been created". And that means
thinking up front about managing the namespace.
-Peff
From: Petr Baudis <hidden> Date: 2016-06-15 22:45:47
On Wed, Dec 17, 2008 at 12:48:02AM +0100, Johannes Schindelin wrote:
Again, do not limit your design by your expectations. People already talk
about having cover letters for their patch series as notes, and Pasky
seems to discuss tracking explicit renames with notes when he does not
play Go, instead of maintaining repo.or.cz and git.or.cz.
I don't really play Go that much anymore! ;-)
Petr "Pasky" Baudis
On Wed, Dec 17, 2008 at 3:38 AM, Jeff King [off-list ref] wrote:
On Tue, Dec 16, 2008 at 12:43:55PM -0600, Govind Salinas wrote:
quoted
I was thinking I would do my first implementation in pyrite and if I find
that it works well I will port it.
OK, though your performance will probably suck unless you dump the notes
tree into a local hash at the beginning of your program. And looking up
every commit's note during revision traversal is one of the intended
uses (e.g., decorating git-log output, or filtering commits based on a
particular note).
Yes, I was thinking that this is the natural way to do things, save that I
would be lazy loading the trees into a cache instead of caching them
all up front. This is one of the reasons that I think the fan out will
help.
And as Dscho mentioned, most of what you need is already there in C.
You are welcome to implement whatever you want in pyrite, of course, but
there is a desire to have this accessible to the revision traversal
machinery. And that means if you want your version in pyrite to be
compatible with what ends up in git, the data structure design needs to
be suitable for both.
Yes, I completely agree that I want it to have the same scheme as what
git will use. That is the reason I posted this here. Since no method
has been formally accepted (checked into master) I wanted to see if
I could nudge things along. I wasn't aware that you and Dscho had
a (very similar) plan. Please, if you guys are decided on the format
then I can just go off and start working on it. But it sounds like there
isn't consensus yet.
<snip>
quoted
root/
12/
34567890123456789012345678901234567890/
abcdef7890123456789012345678901234567890
This way all the notes are attached to a tree, so that gc won't
think they are unreferenced objects.
But you have lost the ordering in your list, then, since they will not
be ordered by sha1 of the note contents. I don't know if you care. The
second sha1 is pointless, anyway, since nobody will know that number as
a reference; why not just name them monotonically starting at 1?
In a later mail I suggested that this be the type or name of the note. Which
I hear is similar to what you suggested.
One of the things I don't like about having several notes is that it
introduces an extra level of indirection that every user has to pay for,
whether they want it or not. If a note can be a blob _or_ a tree, then
those who want to use blobs can reap the performance benefit. Those who
want multiple named notes in a hierarchy can pay the extra indirection
cost.
I haven't measured how big a cost that is (but bearing in mind that we
might want to do this lookup once per revision in a traversal, even one
extra object lookup can have an impact).
That seems reasonable.
I'm also still not convinced the fan-out is worthwhile, but I can see
how it might be. It would be nice to see numbers for both.
<snip>
Also, how large do you expect the list to be under reasonable
quoted
circumstances.
As many notes as there are commits is my goal (e.g., it is not hard to
imagine an automated process to add notes on build status). Ideally, we
could handle as many notes as there are objects; I see no reason not to
allow annotating arbitrary sha1's (I don't know if there is a use for
that, but the more scalable the implementation, the better).
Ah, that is in line with what I was thinking as well.
On naming. I strongly support a ref/notes/sha1/sha1 approach. If
having a type to the note is important, then perhaps the first line of
a note could be considered a type or a set of "tags". This way you
I don't think we are talking about the same thing. What I mean by naming
is "here is a shorthand for referring to notes" that is not necessarily
coupled with the implementation. That is, I would like to do something
like:
git log --notes-filter="foo:bar == 1"
and have that "foo:bar" as a shorthand on each commit for:
refs/notes/foo:$COMMIT/bar
Without a left-hand side (e.g., "bar"), we get:
refs/notes/default:$COMMIT/bar
Or without a right-hand side (e.g., "foo:"), we get:
refs/notes/foo:$COMMIT
I like the overall plan, but I would suggest that --notes[=default] and
--note-type=whatever would be a little friendlier and less error prone.
Thanks for helping me think through this.
-Govind
From: Stephan Beyer <hidden> Date: 2016-06-15 22:45:47
Hi,
Johan Herland wrote:
PS: What's the current status on git-sequencer?
Usable, but I still have a small todo list of things to add
or fix. (And the next days some more time again. I hope it's
enough. Sorry, I'm too lame.)
It's probably the best place to invoke these hooks.
If you want to do stuff on top of git sequencer it's currently
the best to fetch
git://repo.or.cz/git/sbeyer.git seq-builtin-dev
or just to wait. :-\
Regards,
Stephan
--
Stephan Beyer [off-list ref], PGP 0x6EDDD207FCC5040F
From: Jeff King <hidden> Date: 2016-06-15 22:45:47
On Wed, Dec 17, 2008 at 11:06:15AM -0600, Govind Salinas wrote:
Yes, I was thinking that this is the natural way to do things, save that I
would be lazy loading the trees into a cache instead of caching them
all up front. This is one of the reasons that I think the fan out will
help.
I was working under the assumption that you are going to do multiple
note lookups. If you are, then the fan-out isn't really going to help,
as you're going to end up pulling in all of the subtrees anyway. It
helps some if you're only doing a single lookup, but I don't know if
that is measurable.
Yes, I completely agree that I want it to have the same scheme as what
git will use. That is the reason I posted this here. Since no method
has been formally accepted (checked into master) I wanted to see if
I could nudge things along. I wasn't aware that you and Dscho had
a (very similar) plan. Please, if you guys are decided on the format
then I can just go off and start working on it. But it sounds like there
isn't consensus yet.
This is probably not the answer you want, but I think the final design
depends on some C experiments. For example, whether or not there should
be fan-out depends on how it affects performance, which means we need to
do at least partial implementations to compare. So it really is just
waiting for somebody to sit down and do it.
I like the overall plan, but I would suggest that --notes[=default] and
--note-type=whatever would be a little friendlier and less error prone.
But keeping it as a single string means there is no ambiguity when you
specify multiple notes at once. For example:
git log \
--note-filter='test:status == "fail" && importance > 3' \
--pretty=format:%h%n%N(test:errors)
would do something like:
foreach commit $C
compare refs/notes/test:$C/status against the string "fail"
compare refs/notes/default:$C/importance against the number 3
if either don't match, skip the commit
show the hash and the contents of refs/notes/test:$C/errors
and obviously that filter language is totally made up and we may or may
not want to do something that complex. But my point is that we are
defining a namespace of notes, and we want to be able to refer to a
multiple fully qualified names.
-Peff
I redid the benchmark (this time with a bit beefier machine), just
comparing no notes with David's/Peff's idea:
-- snip --
$ GIT_NOTES_TIMING_TESTS=1 sh t3302-notes-index-expensive.sh -i -v
Initialized empty Git repository in /home/gitte/git/t/trash directory.t3302-notes-index-expensive/.git/
* expecting success: create_repo 10
Initialized empty Git repository in /home/gitte/git/t/trash directory.t3302-notes-index-expensive/10/.git/
* ok 1: setup 10
* expecting success: test_notes 10
* ok 2: notes work
* expecting success: time_notes 100
no-notes
0.08user 0.10system 0:00.18elapsed 95%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+58926minor)pagefaults 0swaps
notes
0.14user 0.07system 0:00.54elapsed 38%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+60319minor)pagefaults 0swaps
* ok 3: notes timing
* expecting success: create_repo 100
Initialized empty Git repository in /home/gitte/git/t/trash directory.t3302-notes-index-expensive/100/.git/
* ok 1: setup 100
* expecting success: test_notes 100
* ok 2: notes work
* expecting success: time_notes 100
no-notes
0.23user 0.21system 0:00.45elapsed 96%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+68043minor)pagefaults 0swaps
notes
0.38user 0.21system 0:00.59elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+78829minor)pagefaults 0swaps
* ok 3: notes timing
* expecting success: create_repo 1000
Initialized empty Git repository in /home/gitte/git/t/trash directory.t3302-notes-index-expensive/1000/.git/
* ok 1: setup 1000
* expecting success: test_notes 1000
* ok 2: notes work
* expecting success: time_notes 100
no-notes
2.06user 0.95system 0:04.26elapsed 70%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+159115minor)pagefaults 0swaps
notes
2.83user 1.54system 0:04.38elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+267416minor)pagefaults 0swaps
* ok 3: notes timing
* expecting success: create_repo 10000
Initialized empty Git repository in /home/gitte/git/t/trash directory.t3302-notes-index-expensive/10000/.git/
* ok 1: setup 10000
* expecting success: test_notes 10000
* ok 2: notes work
* expecting success: time_notes 100
no-notes
20.46user 7.63system 0:28.30elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+1083378minor)pagefaults 0swaps
notes
28.78user 13.74system 0:42.85elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+2240296minor)pagefaults 0swaps
* ok 3: notes timing
* passed all 0 test(s)
-- snap --
Keep in mind that the tests run "git log" 99 times, and show the
accumulated time.
So it seems that an increase of roughly 40% in the user time, and roughly
70% in the system time is the price to have notes associated with every
single commit.
Note that in that very same repository, a single "git show" goes from
0.00user 0.00system 0:00.00elapsed 0%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (0major+561minor)pagefaults 0swaps
to this:
0.03user 0.02system 0:00.04elapsed 113%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (0major+2294minor)pagefaults 0swaps
(In another run, it only used 90%CPU)
That's not too shabby, given that Git needs to unpack double the number of
objects in this test when using notes vs. no notes.
For comparison, the numbers back then were something like 10% in user time
with a penalty of an extraordinary magnitude everytime the notes are
updated: around 800%.
Note: all these numbers are worst-case numbers, i.e. every commit has one
note.
To be frank, I do not completely understand why the numbers are that high.
I would have understood an increase roughly 4 seconds for reading the
quite large tree 99 times, and then the same ~0.20 seconds back then.
Maybe I made a huge mistake when implementing the thing.
And BTW, my code does not yet handle the case when
refs/notes/commits:$commit is a tree instead of a blob. That is left as
an exercise to the reader.
Johannes Schindelin (4):
Introduce commit notes
Add a script to edit/inspect notes
Speed up git notes lookup
Add an expensive test for git-notes
.gitignore | 1 +
Documentation/config.txt | 15 ++++
Documentation/git-notes.txt | 46 +++++++++++
Makefile | 3 +
cache.h | 3 +
command-list.txt | 1 +
commit.c | 1 +
config.c | 5 +
environment.c | 1 +
git-notes.sh | 65 +++++++++++++++
notes.c | 159 ++++++++++++++++++++++++++++++++++++++
notes.h | 7 ++
pretty.c | 5 +
t/t3301-notes.sh | 65 +++++++++++++++
t/t3302-notes-index-expensive.sh | 98 +++++++++++++++++++++++
15 files changed, 475 insertions(+), 0 deletions(-)
create mode 100644 Documentation/git-notes.txt
create mode 100755 git-notes.sh
create mode 100644 notes.c
create mode 100644 notes.h
create mode 100755 t/t3301-notes.sh
create mode 100755 t/t3302-notes-index-expensive.sh
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:45:48
Commit notes are blobs which are shown together with the commit
message. These blobs are taken from the notes ref, which you can
configure by the config variable core.notesRef, which in turn can
be overridden by the environment variable GIT_NOTES_REF.
The notes ref is a branch which contains "files" whose names are
the names of the corresponding commits (i.e. the SHA-1).
The rationale for putting this information into a ref is this: we
want to be able to fetch and possibly union-merge the notes,
maybe even look at the date when a note was introduced, and we
want to store them efficiently together with the other objects.
Signed-off-by: Johannes Schindelin <redacted>
---
Documentation/config.txt | 15 ++++++++++
Makefile | 2 +
cache.h | 3 ++
commit.c | 1 +
config.c | 5 +++
environment.c | 1 +
notes.c | 68 ++++++++++++++++++++++++++++++++++++++++++++++
notes.h | 7 +++++
pretty.c | 5 +++
9 files changed, 107 insertions(+), 0 deletions(-)
create mode 100644 notes.c
create mode 100644 notes.h
@@ -431,6 +431,21 @@ core.inithook:: linkgit:git-init[1]. The hook is called with the argument "reinit" if an existing repository is re-initialized.+core.notesRef::+ When showing commit messages, also show notes which are stored in+ the given ref. This ref is expected to contain paths of the form+ ??/*, where the directory name consists of the first two+ characters of the commit name, and the base name consists of+ the remaining 38 characters.+++If such a path exists in the given ref, the referenced blob is read, and+appended to the commit message, separated by a "Notes:" line. If the+given ref itself does not exist, it is not an error, but means that no+notes should be print.+++This setting defaults to "refs/notes/commits", and can be overridden by+the `GIT_NOTES_REF` environment variable.+ alias.*:: Command aliases for the linkgit:git[1] command wrapper - e.g. after defining "alias.last = cat-file commit HEAD", the invocation
@@ -46,6 +46,7 @@ int keep_hard_links = 0;/* Parallel index stat data preload? */intcore_preload_index=0;+char*notes_ref_name;/* This is set by setup_git_dir_gently() and/or git_default_config() */char*git_work_tree_cfg;
@@ -0,0 +1,68 @@+#include"cache.h"+#include"commit.h"+#include"notes.h"+#include"refs.h"+#include"utf8.h"+#include"strbuf.h"++staticintinitialized;++voidget_commit_notes(conststructcommit*commit,structstrbuf*sb,+constchar*output_encoding)+{+staticconstchar*utf8="utf-8";+structstrbufname=STRBUF_INIT;+constchar*hex;+unsignedcharsha1[20];+char*msg;+unsignedlongmsgoffset,msglen;+enumobject_typetype;++if(!initialized){+constchar*env=getenv(GIT_NOTES_REF_ENVIRONMENT);+if(env)+notes_ref_name=getenv(GIT_NOTES_REF_ENVIRONMENT);+elseif(!notes_ref_name)+notes_ref_name=GIT_NOTES_DEFAULT_REF;+if(notes_ref_name&&read_ref(notes_ref_name,sha1))+notes_ref_name=NULL;+initialized=1;+}++if(!notes_ref_name)+return;++strbuf_addf(&name,"%s:%s",notes_ref_name,+sha1_to_hex(commit->object.sha1));+if(get_sha1(name.buf,sha1))+return;++if(!(msg=read_sha1_file(sha1,&type,&msglen))||!msglen||+type!=OBJ_BLOB)+return;++if(output_encoding&&*output_encoding&&+strcmp(utf8,output_encoding)){+char*reencoded=reencode_string(msg,output_encoding,utf8);+if(reencoded){+free(msg);+msg=reencoded;+msglen=strlen(msg);+}+}++/* we will end the annotation by a newline anyway */+if(msglen&&msg[msglen-1]=='\n')+msglen--;++strbuf_addstr(sb,"\nNotes:\n");++for(msgoffset=0;msgoffset<msglen;){+intlinelen=strchrnul(msg,'\n')-msg;++strbuf_addstr(sb," ");+strbuf_add(sb,msg+msgoffset,linelen);+msgoffset+=linelen;+}+free(msg);+}
@@ -0,0 +1,46 @@+git-notes(1)+============++NAME+----+git-notes - Add/inspect commit notes++SYNOPSIS+--------+[verse]+'git-notes' (edit | show) [commit]++DESCRIPTION+-----------+This command allows you to add notes to commit messages, without+changing the commit. To discern these notes from the message stored+in the commit object, the notes are indented like the message, after+an unindented line saying "Notes:".++To disable commit notes, you have to set the config variable+core.notesRef to the empty string. Alternatively, you can set it+to a different ref, something like "refs/notes/bugzilla". This setting+can be overridden by the environment variable "GIT_NOTES_REF".+++SUBCOMMANDS+-----------++edit::+ Edit the notes for a given commit (defaults to HEAD).++show::+ Show the notes for a given commit (defaults to HEAD).+++Author+------+Written by Johannes Schindelin <johannes.schindelin@gmx.de>++Documentation+-------------+Documentation by Johannes Schindelin++GIT+---+Part of the gitlink:git[7] suite
@@ -0,0 +1,65 @@+#!/bin/sh++USAGE="(edit | show) [commit]"+.git-sh-setup++test-n"$3"&&usage++test-z"$1"&&usage+ACTION="$1";shift++test-z"$GIT_NOTES_REF"&&GIT_NOTES_REF="$(gitconfigcore.notesref)"+test-z"$GIT_NOTES_REF"&&GIT_NOTES_REF="refs/notes/commits"++COMMIT=$(gitrev-parse--verify--defaultHEAD"$@")||+die"Invalid commit: $@"++MESSAGE="$GIT_DIR"/new-notes-$COMMIT+trap'+test-f"$MESSAGE"&&rm"$MESSAGE"+'0++case"$ACTION"in+edit)+GIT_NOTES_REF=gitlog-1$COMMIT|sed"s/^/#/">"$MESSAGE"++GIT_INDEX_FILE="$MESSAGE".idx+exportGIT_INDEX_FILE++CURRENT_HEAD=$(gitshow-ref"$GIT_NOTES_REF"|cut-f1-d' ')+if[-z"$CURRENT_HEAD"];then+PARENT=+else+PARENT="-p $CURRENT_HEAD"+gitread-tree"$GIT_NOTES_REF"||die"Could not read index"+gitcat-fileblob:$COMMIT>>"$MESSAGE"2>/dev/null+fi++${VISUAL:-${EDITOR:-vi}}"$MESSAGE"++grep-v^#<"$MESSAGE"|gitstripspace>"$MESSAGE".processed+mv"$MESSAGE".processed"$MESSAGE"+if[-s"$MESSAGE"];then+BLOB=$(githash-object-w"$MESSAGE")||+die"Could not write into object database"+gitupdate-index--add--cacheinfo0644$BLOB$COMMIT||+die"Could not write index"+else+test-z"$CURRENT_HEAD"&&+die"Will not initialise with empty tree"+gitupdate-index--force-remove$COMMIT||+die"Could not update index"+fi++TREE=$(gitwrite-tree)||die"Could not write tree"+NEW_HEAD=$(echoAnnotate$COMMIT|gitcommit-tree$TREE$PARENT)||+die"Could not annotate"+gitupdate-ref-m"Annotate $COMMIT"\+"$GIT_NOTES_REF"$NEW_HEAD$CURRENT_HEAD+;;+show)+gitshow"$GIT_NOTES_REF":$COMMIT+;;+*)+usage+esac
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:45:48
To avoid looking up each and every commit in the notes ref's tree
object, which is very expensive, speed things up by slurping the tree
object's contents into a hash_map.
The idea fo the hashmap singleton is from David Reiss, initial
benchmarking by Jeff King.
Note: the implementation allows for arbitrary entries in the notes
tree object, ignoring those that do not reference a valid object. This
allows you to annotate arbitrary branches, or objects.
Signed-off-by: Johannes Schindelin <redacted>
---
notes.c | 113 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++------
1 files changed, 102 insertions(+), 11 deletions(-)
@@ -4,16 +4,112 @@#include"refs.h"#include"utf8.h"#include"strbuf.h"+#include"tree-walk.h"++structentry{+unsignedcharcommit_sha1[20];+unsignedcharnotes_sha1[20];+};++structhash_map{+structentry*entries;+off_tcount,size;+};staticintinitialized;+staticstructhash_maphash_map;++staticinthash_index(structhash_map*map,constunsignedchar*sha1)+{+inti=((*(unsignedint*)sha1)%map->size);++for(;;){+unsignedchar*current=map->entries[i].commit_sha1;++if(!hashcmp(sha1,current))+returni;++if(is_null_sha1(current))+return-1-i;++if(++i==map->size)+i=0;+}+}++staticvoidadd_entry(constunsignedchar*commit_sha1,+constunsignedchar*notes_sha1)+{+intindex;++if(hash_map.count+1>hash_map.size>>1){+inti,old_size=hash_map.size;+structentry*old=hash_map.entries;++hash_map.size=old_size?old_size<<1:64;+hash_map.entries=(structentry*)+xcalloc(sizeof(structentry),hash_map.size);++for(i=0;i<old_size;i++)+if(!is_null_sha1(old[i].commit_sha1)){+index=-1-hash_index(&hash_map,+old[i].commit_sha1);+memcpy(hash_map.entries+index,old+i,+sizeof(structentry));+}+free(old);+}++index=hash_index(&hash_map,commit_sha1);+if(index<0){+index=-1-index;+hash_map.count++;+}++hashcpy(hash_map.entries[index].commit_sha1,commit_sha1);+hashcpy(hash_map.entries[index].notes_sha1,notes_sha1);+}++staticvoidinitialize_hash_map(constchar*notes_ref_name)+{+unsignedcharsha1[20],commit_sha1[20];+unsigned*mode;+structtree_descdesc;+structname_entryentry;+void*buf;++if(!notes_ref_name||read_ref(notes_ref_name,commit_sha1)||+get_tree_entry(commit_sha1,"",sha1,mode))+return;++buf=fill_tree_descriptor(&desc,sha1);+if(!buf)+die("Could not read %s for notes-index",sha1_to_hex(sha1));++while(tree_entry(&desc,&entry))+if(!get_sha1(entry.path,commit_sha1))+add_entry(commit_sha1,entry.sha1);+free(buf);+}++staticunsignedchar*lookup_notes(constunsignedchar*commit_sha1)+{+intindex;++if(!hash_map.size)+returnNULL;++index=hash_index(&hash_map,commit_sha1);+if(index<0)+returnNULL;+returnhash_map.entries[index].notes_sha1;+}voidget_commit_notes(conststructcommit*commit,structstrbuf*sb,constchar*output_encoding){staticconstchar*utf8="utf-8";-structstrbufname=STRBUF_INIT;-constchar*hex;-unsignedcharsha1[20];+unsignedchar*sha1;char*msg;unsignedlongmsgoffset,msglen;enumobject_typetype;
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:45:48
git-notes have the potential of being pretty expensive, so test with
a lot of commits. A lot. So to make things cheaper, you have to
opt-in explicitely, by setting the environment variable
GIT_NOTES_TIMING_TESTS.
Signed-off-by: Johannes Schindelin <redacted>
---
I would appreciate other people running the tests, and maybe
profiling the code.
However, I will not be really online the next two weeks, so if you
feel like working on this series, go ahead.
Merry Christmas.
t/t3302-notes-index-expensive.sh | 98 ++++++++++++++++++++++++++++++++++++++
1 files changed, 98 insertions(+), 0 deletions(-)
create mode 100755 t/t3302-notes-index-expensive.sh
^^^
Probably should be ${count}.
--
Boyd Stephen Smith Jr. ,= ,-_-. =.
bss@iguanasuicide.net ((_/)o o(\_))
ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-'
http://iguanasuicide.net/ \_/
From: Jeff King <hidden> Date: 2016-06-15 22:45:48
On Sat, Dec 20, 2008 at 12:35:06AM +0100, Johannes Schindelin wrote:
Commit notes are blobs which are shown together with the commit
message. These blobs are taken from the notes ref, which you can
configure by the config variable core.notesRef, which in turn can
be overridden by the environment variable GIT_NOTES_REF.
Hmm. I wanted to try some performance comparisons based on this
implementation, but I can't get your 1/4 to apply. Conflicts in
config.txt and cache.h when applying to master, and "sha1 information is
lacking or useless" for a 3-way merge. What did you base this on?
-Peff
From: Robin Rosenberg <hidden> Date: 2016-06-15 22:45:48
lördag 20 december 2008 07:53:38 skrev Jeff King:
On Sat, Dec 20, 2008 at 12:35:06AM +0100, Johannes Schindelin wrote:
quoted
Commit notes are blobs which are shown together with the commit
message. These blobs are taken from the notes ref, which you can
configure by the config variable core.notesRef, which in turn can
be overridden by the environment variable GIT_NOTES_REF.
Hmm. I wanted to try some performance comparisons based on this
implementation, but I can't get your 1/4 to apply. Conflicts in
config.txt and cache.h when applying to master, and "sha1 information is
lacking or useless" for a 3-way merge. What did you base this on?
patch(1) however can crunch it, with the exception of cache.h. Shouldn't
git am/appy and patch agree on git generated patches (without binary diffs)?
-- robin
From: Jeff King <hidden> Date: 2016-06-15 22:45:48
On Sat, Dec 20, 2008 at 08:55:14AM +0100, Robin Rosenberg wrote:
quoted
Hmm. I wanted to try some performance comparisons based on this
implementation, but I can't get your 1/4 to apply. Conflicts in
config.txt and cache.h when applying to master, and "sha1 information is
lacking or useless" for a 3-way merge. What did you base this on?
patch(1) however can crunch it, with the exception of cache.h. Shouldn't
git am/appy and patch agree on git generated patches (without binary diffs)?
No. git apply is intentionally much more strict about applying under the
assumption that it is better to force a conflict than to silently apply
something that has a reasonable chance of being completely wrong.
And usually it is not a big deal because falling back to the 3-way merge
is a much nicer way of handling any conflicts _anyway_ (I find .rej
files so much more useless than conflict markers, personally).
In this case I was able to:
1. git am /the/patch
2. patch -p1 <.git/rebase-apply/patch
3. manually inspect the results for sanity, and fix up the cache.h
bit that failed totally
4. git add -u && git add notes.[ch]
5. git am --resolved
-Peff
No. It times a git log 100 times (actually, 99 times due to a thinko).
This is only to protect against jitter, otherwise I'd do it only once.
Hth,
Dscho
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:45:48
Commit notes are blobs which are shown together with the commit
message. These blobs are taken from the notes ref, which you can
configure by the config variable core.notesRef, which in turn can
be overridden by the environment variable GIT_NOTES_REF.
The notes ref is a branch which contains "files" whose names are
the names of the corresponding commits (i.e. the SHA-1).
The rationale for putting this information into a ref is this: we
want to be able to fetch and possibly union-merge the notes,
maybe even look at the date when a note was introduced, and we
want to store them efficiently together with the other objects.
Signed-off-by: Johannes Schindelin <redacted>
---
Documentation/config.txt | 15 ++++++++++
Makefile | 2 +
cache.h | 3 ++
commit.c | 1 +
config.c | 5 +++
environment.c | 1 +
notes.c | 68 ++++++++++++++++++++++++++++++++++++++++++++++
notes.h | 7 +++++
pretty.c | 5 +++
9 files changed, 107 insertions(+), 0 deletions(-)
create mode 100644 notes.c
create mode 100644 notes.h
@@ -422,6 +422,21 @@ relatively high IO latencies. With this set to 'true', git will do the index comparison to the filesystem data in parallel, allowing overlapping IO's.+core.notesRef::+ When showing commit messages, also show notes which are stored in+ the given ref. This ref is expected to contain paths of the form+ ??/*, where the directory name consists of the first two+ characters of the commit name, and the base name consists of+ the remaining 38 characters.+++If such a path exists in the given ref, the referenced blob is read, and+appended to the commit message, separated by a "Notes:" line. If the+given ref itself does not exist, it is not an error, but means that no+notes should be print.+++This setting defaults to "refs/notes/commits", and can be overridden by+the `GIT_NOTES_REF` environment variable.+ alias.*:: Command aliases for the linkgit:git[1] command wrapper - e.g. after defining "alias.last = cat-file commit HEAD", the invocation
@@ -45,6 +45,7 @@ enum rebase_setup_type autorebase = AUTOREBASE_NEVER;/* Parallel index stat data preload? */intcore_preload_index=0;+char*notes_ref_name;/* This is set by setup_git_dir_gently() and/or git_default_config() */char*git_work_tree_cfg;
@@ -0,0 +1,68 @@+#include"cache.h"+#include"commit.h"+#include"notes.h"+#include"refs.h"+#include"utf8.h"+#include"strbuf.h"++staticintinitialized;++voidget_commit_notes(conststructcommit*commit,structstrbuf*sb,+constchar*output_encoding)+{+staticconstchar*utf8="utf-8";+structstrbufname=STRBUF_INIT;+constchar*hex;+unsignedcharsha1[20];+char*msg;+unsignedlongmsgoffset,msglen;+enumobject_typetype;++if(!initialized){+constchar*env=getenv(GIT_NOTES_REF_ENVIRONMENT);+if(env)+notes_ref_name=getenv(GIT_NOTES_REF_ENVIRONMENT);+elseif(!notes_ref_name)+notes_ref_name=GIT_NOTES_DEFAULT_REF;+if(notes_ref_name&&read_ref(notes_ref_name,sha1))+notes_ref_name=NULL;+initialized=1;+}++if(!notes_ref_name)+return;++strbuf_addf(&name,"%s:%s",notes_ref_name,+sha1_to_hex(commit->object.sha1));+if(get_sha1(name.buf,sha1))+return;++if(!(msg=read_sha1_file(sha1,&type,&msglen))||!msglen||+type!=OBJ_BLOB)+return;++if(output_encoding&&*output_encoding&&+strcmp(utf8,output_encoding)){+char*reencoded=reencode_string(msg,output_encoding,utf8);+if(reencoded){+free(msg);+msg=reencoded;+msglen=strlen(msg);+}+}++/* we will end the annotation by a newline anyway */+if(msglen&&msg[msglen-1]=='\n')+msglen--;++strbuf_addstr(sb,"\nNotes:\n");++for(msgoffset=0;msgoffset<msglen;){+intlinelen=strchrnul(msg,'\n')-msg;++strbuf_addstr(sb," ");+strbuf_add(sb,msg+msgoffset,linelen);+msgoffset+=linelen;+}+free(msg);+}
@@ -0,0 +1,46 @@+git-notes(1)+============++NAME+----+git-notes - Add/inspect commit notes++SYNOPSIS+--------+[verse]+'git-notes' (edit | show) [commit]++DESCRIPTION+-----------+This command allows you to add notes to commit messages, without+changing the commit. To discern these notes from the message stored+in the commit object, the notes are indented like the message, after+an unindented line saying "Notes:".++To disable commit notes, you have to set the config variable+core.notesRef to the empty string. Alternatively, you can set it+to a different ref, something like "refs/notes/bugzilla". This setting+can be overridden by the environment variable "GIT_NOTES_REF".+++SUBCOMMANDS+-----------++edit::+ Edit the notes for a given commit (defaults to HEAD).++show::+ Show the notes for a given commit (defaults to HEAD).+++Author+------+Written by Johannes Schindelin <johannes.schindelin@gmx.de>++Documentation+-------------+Documentation by Johannes Schindelin++GIT+---+Part of the gitlink:git[7] suite
@@ -0,0 +1,65 @@+#!/bin/sh++USAGE="(edit | show) [commit]"+.git-sh-setup++test-n"$3"&&usage++test-z"$1"&&usage+ACTION="$1";shift++test-z"$GIT_NOTES_REF"&&GIT_NOTES_REF="$(gitconfigcore.notesref)"+test-z"$GIT_NOTES_REF"&&GIT_NOTES_REF="refs/notes/commits"++COMMIT=$(gitrev-parse--verify--defaultHEAD"$@")||+die"Invalid commit: $@"++MESSAGE="$GIT_DIR"/new-notes-$COMMIT+trap'+test-f"$MESSAGE"&&rm"$MESSAGE"+'0++case"$ACTION"in+edit)+GIT_NOTES_REF=gitlog-1$COMMIT|sed"s/^/#/">"$MESSAGE"++GIT_INDEX_FILE="$MESSAGE".idx+exportGIT_INDEX_FILE++CURRENT_HEAD=$(gitshow-ref"$GIT_NOTES_REF"|cut-f1-d' ')+if[-z"$CURRENT_HEAD"];then+PARENT=+else+PARENT="-p $CURRENT_HEAD"+gitread-tree"$GIT_NOTES_REF"||die"Could not read index"+gitcat-fileblob:$COMMIT>>"$MESSAGE"2>/dev/null+fi++${VISUAL:-${EDITOR:-vi}}"$MESSAGE"++grep-v^#<"$MESSAGE"|gitstripspace>"$MESSAGE".processed+mv"$MESSAGE".processed"$MESSAGE"+if[-s"$MESSAGE"];then+BLOB=$(githash-object-w"$MESSAGE")||+die"Could not write into object database"+gitupdate-index--add--cacheinfo0644$BLOB$COMMIT||+die"Could not write index"+else+test-z"$CURRENT_HEAD"&&+die"Will not initialise with empty tree"+gitupdate-index--force-remove$COMMIT||+die"Could not update index"+fi++TREE=$(gitwrite-tree)||die"Could not write tree"+NEW_HEAD=$(echoAnnotate$COMMIT|gitcommit-tree$TREE$PARENT)||+die"Could not annotate"+gitupdate-ref-m"Annotate $COMMIT"\+"$GIT_NOTES_REF"$NEW_HEAD$CURRENT_HEAD+;;+show)+gitshow"$GIT_NOTES_REF":$COMMIT+;;+*)+usage+esac
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:45:48
git-notes have the potential of being pretty expensive, so test with
a lot of commits. A lot. So to make things cheaper, you have to
opt-in explicitely, by setting the environment variable
GIT_NOTES_TIMING_TESTS.
Signed-off-by: Johannes Schindelin <redacted>
---
t/t3302-notes-index-expensive.sh | 98 ++++++++++++++++++++++++++++++++++++++
1 files changed, 98 insertions(+), 0 deletions(-)
create mode 100755 t/t3302-notes-index-expensive.sh
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:45:48
To avoid looking up each and every commit in the notes ref's tree
object, which is very expensive, speed things up by slurping the tree
object's contents into a hash_map.
The idea fo the hashmap singleton is from David Reiss, initial
benchmarking by Jeff King.
Note: the implementation allows for arbitrary entries in the notes
tree object, ignoring those that do not reference a valid object. This
allows you to annotate arbitrary branches, or objects.
Signed-off-by: Johannes Schindelin <redacted>
---
notes.c | 113 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++------
1 files changed, 102 insertions(+), 11 deletions(-)
@@ -4,16 +4,112 @@#include"refs.h"#include"utf8.h"#include"strbuf.h"+#include"tree-walk.h"++structentry{+unsignedcharcommit_sha1[20];+unsignedcharnotes_sha1[20];+};++structhash_map{+structentry*entries;+off_tcount,size;+};staticintinitialized;+staticstructhash_maphash_map;++staticinthash_index(structhash_map*map,constunsignedchar*sha1)+{+inti=((*(unsignedint*)sha1)%map->size);++for(;;){+unsignedchar*current=map->entries[i].commit_sha1;++if(!hashcmp(sha1,current))+returni;++if(is_null_sha1(current))+return-1-i;++if(++i==map->size)+i=0;+}+}++staticvoidadd_entry(constunsignedchar*commit_sha1,+constunsignedchar*notes_sha1)+{+intindex;++if(hash_map.count+1>hash_map.size>>1){+inti,old_size=hash_map.size;+structentry*old=hash_map.entries;++hash_map.size=old_size?old_size<<1:64;+hash_map.entries=(structentry*)+xcalloc(sizeof(structentry),hash_map.size);++for(i=0;i<old_size;i++)+if(!is_null_sha1(old[i].commit_sha1)){+index=-1-hash_index(&hash_map,+old[i].commit_sha1);+memcpy(hash_map.entries+index,old+i,+sizeof(structentry));+}+free(old);+}++index=hash_index(&hash_map,commit_sha1);+if(index<0){+index=-1-index;+hash_map.count++;+}++hashcpy(hash_map.entries[index].commit_sha1,commit_sha1);+hashcpy(hash_map.entries[index].notes_sha1,notes_sha1);+}++staticvoidinitialize_hash_map(constchar*notes_ref_name)+{+unsignedcharsha1[20],commit_sha1[20];+unsigned*mode;+structtree_descdesc;+structname_entryentry;+void*buf;++if(!notes_ref_name||read_ref(notes_ref_name,commit_sha1)||+get_tree_entry(commit_sha1,"",sha1,mode))+return;++buf=fill_tree_descriptor(&desc,sha1);+if(!buf)+die("Could not read %s for notes-index",sha1_to_hex(sha1));++while(tree_entry(&desc,&entry))+if(!get_sha1(entry.path,commit_sha1))+add_entry(commit_sha1,entry.sha1);+free(buf);+}++staticunsignedchar*lookup_notes(constunsignedchar*commit_sha1)+{+intindex;++if(!hash_map.size)+returnNULL;++index=hash_index(&hash_map,commit_sha1);+if(index<0)+returnNULL;+returnhash_map.entries[index].notes_sha1;+}voidget_commit_notes(conststructcommit*commit,structstrbuf*sb,constchar*output_encoding){staticconstchar*utf8="utf-8";-structstrbufname=STRBUF_INIT;-constchar*hex;-unsignedcharsha1[20];+unsignedchar*sha1;char*msg;unsignedlongmsgoffset,msglen;enumobject_typetype;