From: Jeff King <hidden> Date: 2016-06-15 22:45:47
On Wed, Dec 17, 2008 at 04:43:57AM +0100, Johannes Schindelin wrote:
quoted
I agree, I haven't thought of any fix along these lines other than to
make gc do the clean up.
I have, and IIRC I briefly mentioned it back then. Basically, you will
have to add a "git notes gc" or some such, which basically reads in the
whole notes, traverses all reachable commits, marking the corresponding
notes, and then writes out all marked notes (leaving the other notes
behind).
I was thinking something similar, but I think it is even easier. Make
the rule "if we still have the object, then we still have the note".
That has three benefits:
- implementation is simple: for each note $n, delete it unless
has_sha1_file($n).
- it handles notes on non-commit objects
- it kills off notes when an object is _pruned_, not when it stops
being _reachable_. So if I delete a branch with a commit found
nowhere else, its notes will hang around until it is actually pruned.
If I pull it from lost+found, I still keep the notes.
Note that all of this garbage collection of notes is really just
removing them from the most current notes _tree_. If the notes structure
is actually composed of commits, then old notes that are "deleted" will
still be available historically.
I wonder why you speak as if none of that had been implemented yet. From
my work, it is obvious that hashtable is better than sorted list (except
for the fan-out, which obviously wants to be an array), and from Peff's it
is obvious that we want to keep the hashtables in memory.
If he is planning on doing a separate pyrite implementation, then it
_hasn't_ been implemented yet. And I don't care there if he uses
hash tables or sorted lists or whatever. I think the most important
thing is getting down the design of the _data structure_, so that we can
have a compatible implementation inside git itself.
quoted
IMO notes are just a generallized tag.
IMO notes have nothing to do with a tag. Tags point to commits (or other
objects, but that's beside the point here). Notes are pointed _to_ by
commits.
I think maybe we are just talking about semantics, but I think notes are
not pointed to by commits. There is an external mapping of commits to
notes, which is very different. I can give you the commit without you
knowing the notes, or that the notes even exist.
But in practice, I don't know if this distinction is going to influence
any of the design or use.
Has the tree changed? Sure it has. Because Junio committed and pushed
some changes.
I think it is safe to say the tree generally changes for rebase, but not
necessarily for something like an amended commit, or a pull of a patch
sent by mail. So there are times when it changes, and times when it
doesn't.
And if there were some simple way of handling the times when it didn't
change at no general cost, I think going that way would be fine. But:
And the worst part about your idea to attach notes to trees rather than
commits: For things like Acked-by:... you very much want to annotate the
commit, _not_ the tree. The tree is useless here. It says nothing about
the patch, nothing about the explanation, and nothing about the history.
This is a huge cost, IMHO. I think you generally want to annotate
commits, not trees, and the semantic difference is important (but again,
I think all of the proposals are capable of doing either -- but if you
want a "show me the notes on this commit" feature to interoperate, it
needs to pick one).
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:45:47
Hi,
On Wed, 17 Dec 2008, Jeff King wrote:
On Wed, Dec 17, 2008 at 04:43:57AM +0100, Johannes Schindelin wrote:
quoted
quoted
I agree, I haven't thought of any fix along these lines other than
to make gc do the clean up.
I have, and IIRC I briefly mentioned it back then. Basically, you
will have to add a "git notes gc" or some such, which basically reads
in the whole notes, traverses all reachable commits, marking the
corresponding notes, and then writes out all marked notes (leaving the
other notes behind).
I was thinking something similar, but I think it is even easier. Make
the rule "if we still have the object, then we still have the note".
That has three benefits:
- implementation is simple: for each note $n, delete it unless
has_sha1_file($n).
- it handles notes on non-commit objects
- it kills off notes when an object is _pruned_, not when it stops
being _reachable_. So if I delete a branch with a commit found
nowhere else, its notes will hang around until it is actually pruned.
If I pull it from lost+found, I still keep the notes.
Note that all of this garbage collection of notes is really just
removing them from the most current notes _tree_. If the notes structure
is actually composed of commits, then old notes that are "deleted" will
still be available historically.
Right. So my original proposal to use separate refs for separate purposes
might make sense again, so you can have private as well as public notes.
quoted
I wonder why you speak as if none of that had been implemented yet.
From my work, it is obvious that hashtable is better than sorted list
(except for the fan-out, which obviously wants to be an array), and
from Peff's it is obvious that we want to keep the hashtables in
memory.
If he is planning on doing a separate pyrite implementation, then it
_hasn't_ been implemented yet. And I don't care there if he uses hash
tables or sorted lists or whatever. I think the most important thing is
getting down the design of the _data structure_, so that we can have a
compatible implementation inside git itself.
Well, I don't care about pyrite. As far as I am concerned, it might as
well use an incompatible version. I really don't care.
quoted
quoted
IMO notes are just a generallized tag.
IMO notes have nothing to do with a tag. Tags point to commits (or
other objects, but that's beside the point here). Notes are pointed
_to_ by commits.
I think maybe we are just talking about semantics, but I think notes are
not pointed to by commits. There is an external mapping of commits to
notes, which is very different. I can give you the commit without you
knowing the notes, or that the notes even exist.
But in practice, I don't know if this distinction is going to influence
any of the design or use.
You are correct, of course, that the commit does not point to the notes
explicitely, by having a SHA-1 _in_ the commit object. But the main point
still stands: you go from commit to note, not from note to commit. And
this is in stark contrast to tags, where you go from tag to commit, _not_
from commit to tag.
That is a fundamental _difference_ between tags and notes, so that I
refuse to accept the notion of notes being a generalized form of tags.
Ciao,
Dscho
On Wed, Dec 17, 2008 at 4:11 AM, Jeff King [off-list ref] wrote:
On Wed, Dec 17, 2008 at 04:43:57AM +0100, Johannes Schindelin wrote:
quoted
quoted
I agree, I haven't thought of any fix along these lines other than to
make gc do the clean up.
I have, and IIRC I briefly mentioned it back then. Basically, you will
have to add a "git notes gc" or some such, which basically reads in the
whole notes, traverses all reachable commits, marking the corresponding
notes, and then writes out all marked notes (leaving the other notes
behind).
I was thinking something similar, but I think it is even easier. Make
the rule "if we still have the object, then we still have the note".
That has three benefits:
- implementation is simple: for each note $n, delete it unless
has_sha1_file($n).
- it handles notes on non-commit objects
- it kills off notes when an object is _pruned_, not when it stops
being _reachable_. So if I delete a branch with a commit found
nowhere else, its notes will hang around until it is actually pruned.
If I pull it from lost+found, I still keep the notes.
Note that all of this garbage collection of notes is really just
removing them from the most current notes _tree_. If the notes structure
is actually composed of commits, then old notes that are "deleted" will
still be available historically.
This is my concern with keeping a history of the notes pseudo-branch. Let
us say that I do the following
1) on branch A commit a
2) add note a`
3) on branch B commit b
4) add note b`
5) on branch B commit c
6) add note c`
7) delete branch A
8) gc after a time such that a is pruned
Now either I will always have anote a`
Sorry, hit the send key accidentally.
On Wed, Dec 17, 2008 at 4:11 AM, Jeff King [off-list ref] wrote:
On Wed, Dec 17, 2008 at 04:43:57AM +0100, Johannes Schindelin wrote:
quoted
I agree, I haven't thought of any fix along these lines other than to
make gc do the clean up.
I have, and IIRC I briefly mentioned it back then. Basically, you will
quoted
have to add a "git notes gc" or some such, which basically reads in the
whole notes, traverses all reachable commits, marking the corresponding
notes, and then writes out all marked notes (leaving the other notes
behind).
I was thinking something similar, but I think it is even easier. Make
the rule "if we still have the object, then we still have the note".
That has three benefits:
- implementation is simple: for each note $n, delete it unless
has_sha1_file($n).
- it handles notes on non-commit objects
- it kills off notes when an object is _pruned_, not when it stops
being _reachable_. So if I delete a branch with a commit found
nowhere else, its notes will hang around until it is actually pruned.
If I pull it from lost+found, I still keep the notes.
Note that all of this garbage collection of notes is really just
removing them from the most current notes _tree_. If the notes structure
is actually composed of commits, then old notes that are "deleted" will
still be available historically.
This is my concern with keeping a history of the notes pseudo-branch. Let
me restate what you are saying with an example
1) on branch A commit a
2) add note a`
3) on branch B commit b
4) add note b`
5) on branch B commit c
6) add note c`
7) delete branch A
8) gc after a time such that a is pruned
Now either I will always have a note a` as an object forever even though
the only commit that points to it is gone or I have to re-write the history of
the notes branch from the point that it was added.
Given this problem, is it really such a good idea to keep the history?
Of course the other side of this conversation is that the merge operation
will be more complex since the following can also happen
9) push notes
10) user 2 pulls notes but still has commit a and note a`
On the other, other hand, pushing and pulling notes if a history is kept
will have to involve a lot of rebasing/merging.
Just to throw an idea out...
A possible solution is that notes are per-branch,
refs/notes/heads/master
refs/notes/heads/foo/bar
refs/notes/remotes/baz/bang
and then it is easier to deal with. A published branch's notes are isolated
from the changes in unpublished branches. And since published branches
aren't *supposed* to change, then the notes should also always be fast
forwards. Similarly, if a branch is not considered stable, like pu or even
next, then the associated notes branch could be forced in the same way.
Rebase, cherry-pick and merge (and possibly branch/checkout) would have
to be updated to handle notes, which is the down side. It also doesn't solve
the issue of a history causing us to keep notes after the aren't useful anymore.
So perhaps we could use the above layout with no history?
Thanks,
Govind.
On Wed, Dec 17, 2008 at 5:38 AM, Johannes Schindelin
[off-list ref] wrote:
Hi,
On Wed, 17 Dec 2008, Jeff King wrote:
quoted
If he is planning on doing a separate pyrite implementation, then it
_hasn't_ been implemented yet. And I don't care there if he uses hash
tables or sorted lists or whatever. I think the most important thing is
getting down the design of the _data structure_, so that we can have a
compatible implementation inside git itself.
Well, I don't care about pyrite. As far as I am concerned, it might as
well use an incompatible version. I really don't care.
Well I do care. It would not be a good thing for anyone to have 2 separate
systems for notes. Let us say that someone who you work with uses pyrite
and you don't. They will add notes which you can't see and vice versa.
Thanks,
Govind.
From: Jeff King <hidden> Date: 2016-06-15 22:45:48
On Fri, Dec 19, 2008 at 11:38:55AM -0600, Govind Salinas wrote:
This is my concern with keeping a history of the notes pseudo-branch. Let
me restate what you are saying with an example
1) on branch A commit a
2) add note a`
3) on branch B commit b
4) add note b`
5) on branch B commit c
6) add note c`
7) delete branch A
8) gc after a time such that a is pruned
Now either I will always have a note a` as an object forever even though
the only commit that points to it is gone or I have to re-write the history of
the notes branch from the point that it was added.
Yes, that's correct.
Given this problem, is it really such a good idea to keep the history?
I think so. Otherwise how will you push and pull notes? You won't even
know which one is the more recent tree, let alone handle any merges
caused by editing notes in two places.
On the other, other hand, pushing and pulling notes if a history is kept
will have to involve a lot of rebasing/merging.
Depending on your workflow. It might just involve a lot of fast forwards
if the note writer is in one place.
A possible solution is that notes are per-branch,
refs/notes/heads/master
refs/notes/heads/foo/bar
refs/notes/remotes/baz/bang
Sorry, I don't quite get it. You are asking for per-branch notes that
keep history, or per-branch notes that don't keep history?
If the former, then you haven't solved the cruft accumulation problem.
You can get obsolete notes in your note history by rebasing on a branch
that is long-running (which is OK as long as you haven't published
_those particular_ commits). Or are you proposing to rebase and cleanup
the notes history every time you do a destructive operation?
If the latter, then I don't see how you've solved the push-pull and
merge problem (which you need history for).
But in either case, I think the solution is non-intuitive. If I annotate
a commit, and then merge the commit from one branch to another,
shouldn't the annotation stay?
Really, I am not sure this is worth getting too concerned about. Since
we are talking about cruft in the _history_ of the notes branch, it
won't impact actual notes usage (which will always just deal with the
most recent tree). So really we are talking about some uninteresting
objects in the db, which wastes some space. In practice, I suspect this
won't be that large because notes themselves are going to be relatively
short and in many cases, repetitive (i.e., many annotations may have the
same blob hash for several commits). And if it is a space problem, then
the right solution is to periodically truncate the notes history by
rewriting.
-Peff
On Fri, Dec 19, 2008 at 3:25 PM, Jeff King [off-list ref] wrote:
On Fri, Dec 19, 2008 at 11:38:55AM -0600, Govind Salinas wrote:
quoted
This is my concern with keeping a history of the notes pseudo-branch. Let
me restate what you are saying with an example
1) on branch A commit a
2) add note a`
3) on branch B commit b
4) add note b`
5) on branch B commit c
6) add note c`
7) delete branch A
8) gc after a time such that a is pruned
Now either I will always have a note a` as an object forever even though
the only commit that points to it is gone or I have to re-write the history of
the notes branch from the point that it was added.
Yes, that's correct.
quoted
Given this problem, is it really such a good idea to keep the history?
I think so. Otherwise how will you push and pull notes? You won't even
know which one is the more recent tree, let alone handle any merges
caused by editing notes in two places.
Couldn't you simply merge your tree and theirs even if there is no
history. You would have to find a way to handle merges in any event
since they could just as easily happen if you have a history.
quoted
On the other, other hand, pushing and pulling notes if a history is kept
will have to involve a lot of rebasing/merging.
Depending on your workflow. It might just involve a lot of fast forwards
if the note writer is in one place.
quoted
A possible solution is that notes are per-branch,
refs/notes/heads/master
refs/notes/heads/foo/bar
refs/notes/remotes/baz/bang
Sorry, I don't quite get it. You are asking for per-branch notes that
keep history, or per-branch notes that don't keep history?
Both, at the end of my previous mail I said...
"So perhaps we could use the above layout with no history?"
But they are two separate fixes to 2 different problems.
If the former, then you haven't solved the cruft accumulation problem.
You can get obsolete notes in your note history by rebasing on a branch
that is long-running (which is OK as long as you haven't published
_those particular_ commits). Or are you proposing to rebase and cleanup
the notes history every time you do a destructive operation?
Yes, it does not solve that problem. But it does solve things like
Dev1 and Dev2 both have branches A and topic branch B. and they
are in refs/notes/public (or refs/notes or something not branch specific).
Dev1 adds 100 notes to topic B, lets say half of them are obsolete due
to rebases or whatever. Dev2 pulls A and updates their notes
as well. Now Dev2 has acquired all the notes from Dev1 including the
obsolete ones. So you have 100 commits, 100 blobs and all the new
trees that go with them that the user was not interested in.
Run this across 1000 users and you have a lot of cruft.
Now, if instead we have a per-branch notes scheme, then you only get
the cruft from the branches you were interested in. If you remove the
history you could end up with no cruft because gc should handle it.
If the latter, then I don't see how you've solved the push-pull and
merge problem (which you need history for).
What git-fetch would have to do is say. This is a note. The remote
sha is not the same as mine, i will treat this as a force and fetch the
objects without checking history and then run a merge on the 2
commits. The notes merge could have its own strategy that checked
if an object exists before deciding to add a new item or delete a
removed one. Then the user would only have to intervene if the
notes where edited.
But in either case, I think the solution is non-intuitive. If I annotate
a commit, and then merge the commit from one branch to another,
shouldn't the annotation stay?
Sure, either the merge command could run 2 merges, one for the
real branch and one for the notes pseudo branch or the user
could be required to do that manually. I would think that doing
it automatically would be good. Especially if you use a special
merge strategy.
Really, I am not sure this is worth getting too concerned about. Since
we are talking about cruft in the _history_ of the notes branch, it
won't impact actual notes usage (which will always just deal with the
most recent tree). So really we are talking about some uninteresting
objects in the db, which wastes some space. In practice, I suspect this
won't be that large because notes themselves are going to be relatively
short and in many cases, repetitive (i.e., many annotations may have the
same blob hash for several commits). And if it is a space problem, then
the right solution is to periodically truncate the notes history by
rewriting.
You are correct of course that it will just be wasted space. But I am
concerned that it could end up being a lot of wasted space. I mean, what
if every person who contributed to the kernel contributed note cruft. Users
have branches that they consider public, so they might go into the a public
note store if there is no per-branch store. Or errant users could use the
public store without understanding how they are affecting the central repo,
including the obsolete ones.
If you *really* don't think its something to be worried about then I am OK
with that since you have a lot more experience with this, but it sounds hairy
to me.
Thanks,
Govind.
From: Jeff King <hidden> Date: 2016-06-15 22:45:48
On Fri, Dec 19, 2008 at 04:24:01PM -0600, Govind Salinas wrote:
quoted
I think so. Otherwise how will you push and pull notes? You won't even
know which one is the more recent tree, let alone handle any merges
caused by editing notes in two places.
Couldn't you simply merge your tree and theirs even if there is no
history. You would have to find a way to handle merges in any event
since they could just as easily happen if you have a history.
Let's say I have a tree T1 like this:
$COMMIT_A -> $BLOB_A
$COMMIT_B -> $BLOB_B1
and a tree T2 like this:
$COMMIT_B -> $BLOB_B2
$COMMIT_C -> $BLOB_C
what is the correct merge? Was $COMMIT_A added in T1, or deleted in T2?
How about $COMMIT_C? Even if you went with a strategy like "always add
from both" (which I don't think is a good idea, because deleted notes
will keep popping back up) you have a conflict with $COMMIT_B. Should
it be B1 or B2? You can't tell if B1 became B2, vice versa, or if there
is a true merge conflict.
quoted
If the former, then you haven't solved the cruft accumulation problem.
You can get obsolete notes in your note history by rebasing on a branch
that is long-running (which is OK as long as you haven't published
_those particular_ commits). Or are you proposing to rebase and cleanup
the notes history every time you do a destructive operation?
Yes, it does not solve that problem. But it does solve things like
Dev1 and Dev2 both have branches A and topic branch B. and they
are in refs/notes/public (or refs/notes or something not branch specific).
Dev1 adds 100 notes to topic B, lets say half of them are obsolete due
to rebases or whatever. Dev2 pulls A and updates their notes
as well. Now Dev2 has acquired all the notes from Dev1 including the
obsolete ones. So you have 100 commits, 100 blobs and all the new
trees that go with them that the user was not interested in.
Run this across 1000 users and you have a lot of cruft.
Now, if instead we have a per-branch notes scheme, then you only get
the cruft from the branches you were interested in. If you remove the
history you could end up with no cruft because gc should handle it.
OK. But my point is that this is an incomplete solution. You can _still_
get cruft, and you _still_ have to deal with that cruft some other way.
So we will still end up having to implement something else. And I might
even be fine with a partial solution that helped some if it didn't come
with a cost, but I think the "notes stick to branches" behavior is
strictly worse.
quoted
If the latter, then I don't see how you've solved the push-pull and
merge problem (which you need history for).
What git-fetch would have to do is say. This is a note. The remote
sha is not the same as mine, i will treat this as a force and fetch the
objects without checking history and then run a merge on the 2
commits. The notes merge could have its own strategy that checked
if an object exists before deciding to add a new item or delete a
removed one. Then the user would only have to intervene if the
notes where edited.
I don't like that because:
- the user is going to end up manually resolving merge conflicts for
things that _should_ have been fast forwards. But much worse, it's
going to be on content they may never even have seen before. How
will they decide which is which?
- how do you push notes? There's no opportunity to handle the merge
on the remote side. And you can't just pull, merge locally, and push
what is now a fast-forward, because there is no concept of
fast-forward without history.
- Suddenly pulling and pushing notes isn't just taken care of by the
usual ref transfer mechanisms. We have to implement a whole new
system.
You are correct of course that it will just be wasted space. But I am
concerned that it could end up being a lot of wasted space. I mean, what
if every person who contributed to the kernel contributed note cruft. Users
What if every person who contributed to the kernel contributed history
cruft? It's really the same problem, and it is solved by people keeping
their trees clean (via rebase) and being picky about how data comes into
your tree (i.e., don't pull from people with cruft). I suspect Linus
wouldn't pull notes at all (and they wouldn't make it over patch
transmission anyway). But in a workflow that is pulling the notes, the
right time to clean up history is probably before publishing. That is,
you can rebase and clean up your notes history just before you push it
to somewhere public, just like you might clean up messy history.
If you *really* don't think its something to be worried about then I
am OK with that since you have a lot more experience with this, but it
sounds hairy to me.
It is hairy, and I wish there were a better solution. But I think every
other option is much worse.
-Peff