From: Junio C Hamano <hidden> Date: 2016-06-15 22:52:33
Jeff King [off-list ref] writes:
When the objects become unreferenced, we eject them from the pack into
loose form again. If they don't become referenced in the 2-week window,
they get pruned then. So yes, you drop the age information, but they do
eventually go away.
If you update gc/repack -A to put them in a separate pack, then you would
never be able to get rid of them, no? You pack, then eject (which gives
them a fresher timestamp), then notice that you are within the 2-week window
and pack them again,...
From: Jeff King <hidden> Date: 2016-06-15 22:52:33
On Fri, Dec 02, 2011 at 09:35:52AM -0800, Junio C Hamano wrote:
Jeff King [off-list ref] writes:
quoted
When the objects become unreferenced, we eject them from the pack into
loose form again. If they don't become referenced in the 2-week window,
they get pruned then. So yes, you drop the age information, but they do
eventually go away.
If you update gc/repack -A to put them in a separate pack, then you would
never be able to get rid of them, no? You pack, then eject (which gives
them a fresher timestamp), then notice that you are within the 2-week window
and pack them again,...
But we shouldn't be packing totally unreferenced objects. Barring bugs,
the life cycle of such an object should be something like:
1. Object X is created on branch 'foo'.
2. Branch 'foo' is deleted, but its commits are still in the HEAD
reflog, referencing X.
3. 90 days pass (actually, I think this might be the 30-day
expire-unreachable time)
4. "git gc" runs "git repack -Ad", which will eject X from the pack
into a loose form (because it is not becoming part of the new pack
we are writing).
5. Two weeks pass.
6. "git gc" runs "git prune --expire=2.weeks.ago", which removes the
object.
"gc" runs between (4) and (6) will not re-pack the object, because it
remains unreferenced.
I think things might be slowed somewhat by "gc --auto", which will not
do a "repack -A" until we have too many packs. So steps (3) and (4) are
really more like "gc runs git-repack without -A" 50 times, and then we
finally run "git repack -A".
-Peff
On Fri, Dec 2, 2011 at 11:45 AM, Jeff King [off-list ref] wrote:
On Fri, Dec 02, 2011 at 09:35:52AM -0800, Junio C Hamano wrote:
quoted
Jeff King [off-list ref] writes:
quoted
When the objects become unreferenced, we eject them from the pack into
loose form again. If they don't become referenced in the 2-week window,
they get pruned then. So yes, you drop the age information, but they do
eventually go away.
If you update gc/repack -A to put them in a separate pack, then you would
never be able to get rid of them, no? You pack, then eject (which gives
them a fresher timestamp), then notice that you are within the 2-week window
and pack them again,...
But we shouldn't be packing totally unreferenced objects. Barring bugs,
the life cycle of such an object should be something like:
1. Object X is created on branch 'foo'.
2. Branch 'foo' is deleted, but its commits are still in the HEAD
reflog, referencing X.
3. 90 days pass (actually, I think this might be the 30-day
expire-unreachable time)
4. "git gc" runs "git repack -Ad", which will eject X from the pack
into a loose form (because it is not becoming part of the new pack
we are writing).
Actually, it is right here when the newly loosened unreferenced
objects will be deleted. Objects ejected from a pack _are_ given the
timestamp of the pack they were ejected from. So, if the pack is
older than two weeks (90 days in your example), then so will be the
loosened objects, and git prune will delete them when called by git
gc.
5. Two weeks pass.
6. "git gc" runs "git prune --expire=2.weeks.ago", which removes the
object.
"gc" runs between (4) and (6) will not re-pack the object, because it
remains unreferenced.
Correct with the recognition that loose objects get pack mtime, so
step 5 may be less than two weeks.
I think things might be slowed somewhat by "gc --auto", which will not
do a "repack -A" until we have too many packs. So steps (3) and (4) are
really more like "gc runs git-repack without -A" 50 times, and then we
finally run "git repack -A".
This is correct. This should have the effect of increasing the age of
unreferenced objects when they are finally loosened and make it more
likely that they are pruned during the same git gc operation that
loosens them.
Linus's scenario of fetching a lot of stuff that never actually makes
it into the reflogs is still a valid problem. I'm not sure that
people who don't know what they are doing are going to run into this
problem though. Since he fetches a lot of stuff without ever checking
it out or creating a branch from it, potentially many objects become
unreferenced every time FETCH_HEAD changes. If he does this many
times in a short period of time, he could reach the gc.autopacklimit
and trigger gc --auto and produce more than gc.auto loose objects that
are younger than gc.pruneExpire.
Decreasing gc.pruneExpire as you suggested should make it much less
likely to run into this problem. I wonder if it is worth trying to
limit how often gc --auto is run to not be more often than
gc.pruneExpire or something. If we modified the timestamp that is
assigned to fetched packs, maybe we could use the pack timestamps as
an indicator of how recently git gc has run.
-Brandon
From: Nicolas Pitre <nico@fluxnic.net> Date: 2016-06-15 22:52:34
On Sat, 3 Dec 2011, Brandon Casey wrote:
Linus's scenario of fetching a lot of stuff that never actually makes
it into the reflogs is still a valid problem. I'm not sure that
people who don't know what they are doing are going to run into this
problem though. Since he fetches a lot of stuff without ever checking
it out or creating a branch from it, potentially many objects become
unreferenced every time FETCH_HEAD changes.
Maybe FETCH_HEAD should have a reflog too?
Nicolas
From: Jeff King <hidden> Date: 2016-06-15 22:52:35
On Wed, Dec 07, 2011 at 05:12:14PM -0500, Nicolas Pitre wrote:
Maybe FETCH_HEAD should have a reflog too?
That might be nice. However, there is a complication, in that FETCH_HEAD
may contain many sha1s, but each reflog entry only has room for a single
sha1 transition. You could obviously encode it as a series of reflog
entries, but then "git show FETCH_HEAD@{1}" is not very meaningful.
-Peff
From: Nicolas Pitre <nico@fluxnic.net> Date: 2016-06-15 22:52:35
On Wed, 7 Dec 2011, Jeff King wrote:
On Wed, Dec 07, 2011 at 05:12:14PM -0500, Nicolas Pitre wrote:
quoted
Maybe FETCH_HEAD should have a reflog too?
That might be nice. However, there is a complication, in that FETCH_HEAD
may contain many sha1s, but each reflog entry only has room for a single
sha1 transition. You could obviously encode it as a series of reflog
entries, but then "git show FETCH_HEAD@{1}" is not very meaningful.
What does "git show FETCH_HEAD" do now? If it shows only one
(presumably the first) SHA1 then its reflog doesn't have to be smarter,
which would properly cover most cases already. I certainly never did a
multi-ref fetch myself.
Nicolas
From: Jeff King <hidden> Date: 2016-06-15 22:52:35
On Wed, Dec 07, 2011 at 07:18:13PM -0500, Nicolas Pitre wrote:
quoted
quoted
Maybe FETCH_HEAD should have a reflog too?
That might be nice. However, there is a complication, in that FETCH_HEAD
may contain many sha1s, but each reflog entry only has room for a single
sha1 transition. You could obviously encode it as a series of reflog
entries, but then "git show FETCH_HEAD@{1}" is not very meaningful.
What does "git show FETCH_HEAD" do now? If it shows only one
(presumably the first) SHA1 then its reflog doesn't have to be
smarter, which would properly cover most cases already.
Are you proposing that it only store the first ref in the reflog, or
that we accept that a single fetch may write lots of reflog entries?
If the former, then you are missing the expiration/connectivity
properties.
If the latter, then it is not just "we only show the first one for
FETCH_HEAD@{1}", but also "the thing that used to be FETCH_HEAD@{1} does
not graduate to FETCH_HEAD@{2}, but rather FETCH_HEAD@{n} for some
unknown n". That may be an acceptable limitation; I just wanted to
mention it in case somebody can think of some clever solution.
I certainly never did a multi-ref fetch myself.
Not consciously, perhaps, but you do it all the time without realizing
it:
$ git clone git://git.kernel.org/pub/scm/git/git.git
$ cd git
$ git fetch -v origin
= [up to date] maint -> origin/maint
= [up to date] master -> origin/master
= [up to date] next -> origin/next
= [up to date] pu -> origin/pu
= [up to date] todo -> origin/todo
$ cat .git/FETCH_HEAD
b1af9630d758e1728fc0008b3f18d90d8f87f4c5 not-for-merge branch 'maint' of git://git.kernel.org/pub/scm/git/git
4cb5d10b14dcbe0155bed9c45ccb94e83bd4c599 branch 'master' of git://git.kernel.org/pub/scm/git/git
03e5527c5df33d4550ccc1446d861c0aa5689d58 not-for-merge branch 'next' of git://git.kernel.org/pub/scm/git/git
cc4e3f01fc6a5e09ae5bbdc464965981fae4cf39 not-for-merge branch 'pu' of git://git.kernel.org/pub/scm/git/git
7a02dba15bd28826344f9c14a5e2b5c57eeb7e50 not-for-merge branch 'todo' of git://git.kernel.org/pub/scm/git/git
-Peff
From: Jeff King <hidden> Date: 2016-06-15 22:52:35
On Sat, Dec 03, 2011 at 01:42:22PM -0600, Brandon Casey wrote:
quoted
4. "git gc" runs "git repack -Ad", which will eject X from the pack
into a loose form (because it is not becoming part of the new pack
we are writing).
Actually, it is right here when the newly loosened unreferenced
objects will be deleted. Objects ejected from a pack _are_ given the
timestamp of the pack they were ejected from. So, if the pack is
older than two weeks (90 days in your example), then so will be the
loosened objects, and git prune will delete them when called by git
gc.
Thanks, I didn't notice that when looking at the code.
Decreasing gc.pruneExpire as you suggested should make it much less
likely to run into this problem.
I'd be more comfortable with that solution if we had data on what the
timestamps look like when it actually happens (e.g., an "ls -lR" listing
of a repository that in practice is wanting to auto-gc too often).
I wonder if it is worth trying to limit how often gc --auto is run to
not be more often than gc.pruneExpire or something. If we modified
the timestamp that is assigned to fetched packs, maybe we could use
the pack timestamps as an indicator of how recently git gc has run.
I'm worried you run into other corner cases, there. Like a repository
which is generating new, referenced objects at a fast rate (e.g.,
because you're importing something) should trigger auto-gc much sooner
than that, and this rule would prevent it.
-Peff
From: Nicolas Pitre <nico@fluxnic.net> Date: 2016-06-15 22:52:35
On Wed, 7 Dec 2011, Jeff King wrote:
On Wed, Dec 07, 2011 at 07:18:13PM -0500, Nicolas Pitre wrote:
quoted
I certainly never did a multi-ref fetch myself.
Not consciously, perhaps, but you do it all the time without realizing
it:
$ git clone git://git.kernel.org/pub/scm/git/git.git
$ cd git
$ git fetch -v origin
= [up to date] maint -> origin/maint
= [up to date] master -> origin/master
= [up to date] next -> origin/next
= [up to date] pu -> origin/pu
= [up to date] todo -> origin/todo
$ cat .git/FETCH_HEAD
b1af9630d758e1728fc0008b3f18d90d8f87f4c5 not-for-merge branch 'maint' of git://git.kernel.org/pub/scm/git/git
4cb5d10b14dcbe0155bed9c45ccb94e83bd4c599 branch 'master' of git://git.kernel.org/pub/scm/git/git
03e5527c5df33d4550ccc1446d861c0aa5689d58 not-for-merge branch 'next' of git://git.kernel.org/pub/scm/git/git
cc4e3f01fc6a5e09ae5bbdc464965981fae4cf39 not-for-merge branch 'pu' of git://git.kernel.org/pub/scm/git/git
7a02dba15bd28826344f9c14a5e2b5c57eeb7e50 not-for-merge branch 'todo' of git://git.kernel.org/pub/scm/git/git
OK, nevermind. I admitedly never have been close enough to the related
code.
And I don't think this particular case is interesting anyway as the
reflogs for the various branches alre already involved. I was thinking
more about the "git fetch git://some.random.repo foobar" case where the
summary also explicitly shows:
From: git://some.random.repo
...... foobar -> FETCH_HEAD
In that case the only reference to the fetched branch is stored in
FETCH_HEAD and that is what might be worthwile for a reflog.
Nicolas
From: Jeff King <hidden> Date: 2016-06-15 22:52:35
On Wed, Dec 07, 2011 at 10:35:00PM -0500, Nicolas Pitre wrote:
And I don't think this particular case is interesting anyway as the
reflogs for the various branches alre already involved. I was thinking
more about the "git fetch git://some.random.repo foobar" case where the
summary also explicitly shows:
From: git://some.random.repo
...... foobar -> FETCH_HEAD
In that case the only reference to the fetched branch is stored in
FETCH_HEAD and that is what might be worthwile for a reflog.
I agree that is the interesting case. Perhaps we could just not bother
writing the other case into the reflog at all. So the reflog would be
sensible and contain only the set of things they had fetched or pulled
explicitly by URL. If they really want to do a multi-ref one-off fetch
from some URL, then we write multiple reflog entries. But at least the
user is very aware of what they've done, so they're not surprised by the
reflog advancing by more than 1 entry.
-Peff