I know we tried this once earlier, and it caused problems, but that was
when pack-files were new, and not everybody could handle them. These days,
if you can't handle pack-files, kernel.org is already pretty useless,
because all the major packages use them anyway, because people have
packed their repositories by hand.
So I'm suggesting we try to do an automatic repack every once in a while.
In my suggestion, there would be two levels of repacking: "incremental"
and "full", and both of them would count the number of files before they
run, so that you'd only do it when it seems worthwhile.
This is a _really_ simple heuristic:
- incremental repacking run every day:
#
# Check if we have more than a couple of hundred
# unpacked objects - approximated by whether we
# have any "00" directory with more than one
#
# This means that we don't repack projects that
# that don't have a lot of work going on.
#
# Note: with really new versions of git, the "00"
# directory may not exist if it has been pruned
# away, so handle that gracefully.
#
export GIT_DIR=${1:-.}
objs=$(find "$GIT_DIR/objects/00" -type f 2> /dev/null | wc -l)
if [ "$obj" -gt 0 ]; then
git repack &&
git prune-packed
fi
- "full repack" every week if the number of packs has grown to be bigger
than say 10 (ie even a very active projects will never have a full
repack more than every other week)
#
# Check if we have lots of packs, where "lots" is defined as 10.
#
# Note: with something that was generated with an old version
# of git, the "pack" directory may not exist, so handle that
# gracefully.
#
export GIT_DIR=${1:-.}
packs=$(find "$GIT_DIR/objects/pack" -name '*.idx' 2> /dev/null | wc -l)
if [ "$packs" -gt 10 ]; then
git repack -a -d &&
git prune-packed
fi
- do a full repack of everything once to start with.
export GIT_DIR=${1:-.}
git repack -a -d &&
git prune-packed
the above three trivial scripts just take a single argument, which becomes
the GIT_DIR (and if no argument exists, it would default to ".")
Is there any reason not to do this? Right now mirroring is slow, and
webgit is also getting to be very slow sometimes. I bet we'd be _much_
better off with this kind of setup.
NOTE! The above is the "stupid" approach, which totally ignores alternate
directories, and isn't able to take advantage of the fact that many
projects could share objects. But it's simple, and it's efficient (eg it
won't spend time on things like the large historic archives which don't
change, but that would be expensive to repack if you didn't check for the
need).
So we could try to come up with a better approach eventually, which would
automatically notice alternate directories and not repack stuff that
exists there, but I'm pretty sure that the above would already help a
_lot_, and while pack-files have been been around forever, the
"alternates" support is still pretty new, so the above is also the "safer"
thing to do.
We'd only do the automatic thing on stuff under /pub/scm, of course: not
stuff in peoples home directories etc..
Peter?
Linus
From: Carl Baldwin <hidden> Date: 2016-06-15 22:42:13
I have a question about automatic repacking.
I am thinking of turning something like Linus' repacking heuristic loose
on my repositories. I just want to make sure it is as safe as possible.
At the core of the incremental and full repack strategies are these
statements.
Incremental...
git repack &&
git prune-packed
Full...
git repack -a -d &&
git prune-packed
Are there some built in safety checks in 'git repack' and/or 'git
prune-packed' to guard against corruption? In the long run, I would
feel more comfortable with somelike like this:
git repack
git verify-pack <new pack>
git prune-packed
Would something like this even work with 'git repack -a -d'? Is there a
way to do something like the following for a full repack to achieve the
ultimate in paranoia?
git repack -a
git verify-pack <new pack file>
git trash-redundant-packs <new pack file>
git prune-packed
Carl
On Thu, Oct 13, 2005 at 11:44:30AM -0700, Linus Torvalds wrote:
I know we tried this once earlier, and it caused problems, but that was
when pack-files were new, and not everybody could handle them. These days,
if you can't handle pack-files, kernel.org is already pretty useless,
because all the major packages use them anyway, because people have
packed their repositories by hand.
So I'm suggesting we try to do an automatic repack every once in a while.
In my suggestion, there would be two levels of repacking: "incremental"
and "full", and both of them would count the number of files before they
run, so that you'd only do it when it seems worthwhile.
This is a _really_ simple heuristic:
- incremental repacking run every day:
#
# Check if we have more than a couple of hundred
# unpacked objects - approximated by whether we
# have any "00" directory with more than one
#
# This means that we don't repack projects that
# that don't have a lot of work going on.
#
# Note: with really new versions of git, the "00"
# directory may not exist if it has been pruned
# away, so handle that gracefully.
#
export GIT_DIR=${1:-.}
objs=$(find "$GIT_DIR/objects/00" -type f 2> /dev/null | wc -l)
if [ "$obj" -gt 0 ]; then
git repack &&
git prune-packed
fi
- "full repack" every week if the number of packs has grown to be bigger
than say 10 (ie even a very active projects will never have a full
repack more than every other week)
#
# Check if we have lots of packs, where "lots" is defined as 10.
#
# Note: with something that was generated with an old version
# of git, the "pack" directory may not exist, so handle that
# gracefully.
#
export GIT_DIR=${1:-.}
packs=$(find "$GIT_DIR/objects/pack" -name '*.idx' 2> /dev/null | wc -l)
if [ "$packs" -gt 10 ]; then
git repack -a -d &&
git prune-packed
fi
- do a full repack of everything once to start with.
export GIT_DIR=${1:-.}
git repack -a -d &&
git prune-packed
the above three trivial scripts just take a single argument, which becomes
the GIT_DIR (and if no argument exists, it would default to ".")
Is there any reason not to do this? Right now mirroring is slow, and
webgit is also getting to be very slow sometimes. I bet we'd be _much_
better off with this kind of setup.
NOTE! The above is the "stupid" approach, which totally ignores alternate
directories, and isn't able to take advantage of the fact that many
projects could share objects. But it's simple, and it's efficient (eg it
won't spend time on things like the large historic archives which don't
change, but that would be expensive to repack if you didn't check for the
need).
So we could try to come up with a better approach eventually, which would
automatically notice alternate directories and not repack stuff that
exists there, but I'm pretty sure that the above would already help a
_lot_, and while pack-files have been been around forever, the
"alternates" support is still pretty new, so the above is also the "safer"
thing to do.
We'd only do the automatic thing on stuff under /pub/scm, of course: not
stuff in peoples home directories etc..
Peter?
Linus
-
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
I have a question about automatic repacking.
I am thinking of turning something like Linus' repacking heuristic loose
on my repositories. I just want to make sure it is as safe as possible.
At the core of the incremental and full repack strategies are these
statements.
Incremental...
quoted
git repack &&
git prune-packed
Full...
quoted
git repack -a -d &&
git prune-packed
NOTE! Since that email, "git repack" has gotten a "local" option (-l),
which is very useful if the repositories have pointers to alternates.
So do
git repack -l
instead, to get much better packs (and "-a -d" for the full case, of
course).
Other that than, the old email suggestion should still be fine.
Are there some built in safety checks in 'git repack' and/or 'git
prune-packed' to guard against corruption? In the long run, I would
feel more comfortable with somelike like this:
git repack
git verify-pack <new pack>
git prune-packed
You can certainly do that if you are nervous. It might even be a good
idea: just for fun, I just did
git clone -l git git-clone
cd git-clone
# pick an object at random
rm .git/objects/f7/c3d39fe3db6da3a307da385a7a1cb563ed15f7
git repack -a -d
and it said:
error: Could not read f7c3d39fe3db6da3a307da385a7a1cb563ed15f7
fatal: bad tree object f7c3d39fe3db6da3a307da385a7a1cb563ed15f7
but then it created the pack _anyway_, and said:
Packing 27 objects
Pack pack-13bfca704078175c1c1c59964553b14f7b952651 created.
and happily removed all the old ones.
So right now, repacking a broken archive can actually break it even more.
NOTE! Your "git verify-pack" wouldn't even catch this: the _pack_ is fine,
it's just incomplete.
Of course, this only happens if the repository was broken to begin with,
so arguably it's not that bad. But it does show that git-repack should be
more careful and return an error more aggressively.
Can anybody tell me how to do that sanely? Right now we do
..
name=$(git-rev-list --objects $rev_list $(git-rev-parse $rev_parse) |
git-pack-objects --non-empty $pack_objects .tmp-pack) ||
exit 1
..
and the thing is, the "git-pack-objects" thing is happy, it's the
"git-rev-list" that fails. So because the last command in the pipeline
returns ok, we think it all is ok..
(This is one of the reasons I much prefer working in C over working in
shell: it may be twenty times more lines, but when you have a problem, the
fix is always obvious..)
Anyway, with that fixed, a "git repack" in many ways would be a mini-fsck,
so it should be very safe in general. Modulo any other bugs like the
above.
Linus
From: Chuck Lever <hidden> Date: 2016-06-15 22:42:13
Linus Torvalds wrote:
On Mon, 21 Nov 2005, Carl Baldwin wrote:
quoted
I have a question about automatic repacking.
I am thinking of turning something like Linus' repacking heuristic loose
on my repositories. I just want to make sure it is as safe as possible.
At the core of the incremental and full repack strategies are these
statements.
Incremental...
quoted
git repack &&
git prune-packed
Full...
quoted
git repack -a -d &&
git prune-packed
NOTE! Since that email, "git repack" has gotten a "local" option (-l),
which is very useful if the repositories have pointers to alternates.
So do
git repack -l
instead, to get much better packs (and "-a -d" for the full case, of
course).
Other that than, the old email suggestion should still be fine.
i've been playing with "git repack" on StGIT-managed repositories.
on NFS, using packs instead of individual objects is quite a bit faster,
because a single NFS GETATTR will tell you if your NFS client's cached
pack file is still valid, whereas a whole bunch of GETATTRs are required
for validating individual object files.
there are some things repacking does that breaks StGIT, though.
git repack -d
seems to remove old commits that StGIT was still depending on.
git repack -a -n
seems to work fine with StGIT, as does
git prune-packed
i'm really interested in trying out the new command to remove redundant
objects and packs, but haven't gotten around to it yet.
there are some things repacking does that breaks StGIT, though.
git repack -d
seems to remove old commits that StGIT was still depending on.
If that is true, then "git-fsck-cache" probably also reports errors on a
StGIT repository. No? Basically, it implies that the tool doesn't know how
to find all the "heads".
Could somebody (Catalin?) perhaps tell how tools like git-fsck-cache and
git-repack could figure out which objects are still in use by stgit?
Preferably with some generic mechanism that _other_ projects (not just
stgit) might want to use?
The preferred way would be to just list the references somewhere under
.git/refs/stgit, in which case fsck and repack should pick them up
automatically (so clearly stgit doesn't do that right now ;).
It also implies that doing a "git prune" will do horribly bad things to a
stgit repo, since it would remove all the objects that it thinks aren't
reachable..
git repack -a -n
seems to work fine with StGIT,
Well, it "works", but not "fine". Since it doesn't know about the stgit
objects, it won't ever pack them.
But maybe that's what stgit wants (since they are "temporary"), but it
does mean that if you see a big advantage from packing, you might be
losing some of it.
Linus
there are some things repacking does that breaks StGIT, though.
git repack -d
seems to remove old commits that StGIT was still depending on.
If that is true, then "git-fsck-cache" probably also reports errors on a
StGIT repository. No? Basically, it implies that the tool doesn't know how
to find all the "heads".
Indeed, 'git repack -d' or 'git prune' might remove the patches which
are not applied since there is no link to them from .git/refs/.
Could somebody (Catalin?) perhaps tell how tools like git-fsck-cache and
git-repack could figure out which objects are still in use by stgit?
They don't figure this out at the moment. I initially thought about
implementing these commands in StGIT so that they would pass the
proper references.
Preferably with some generic mechanism that _other_ projects (not just
stgit) might want to use?
The preferred way would be to just list the references somewhere under
.git/refs/stgit, in which case fsck and repack should pick them up
automatically (so clearly stgit doesn't do that right now ;).
I thought about adding .git/refs/patches/<branch>/* files
corresponding to the every StGIT patch. Are the above git commands
looking at all depths in the .git/refs/ directory?
quoted
git repack -a -n
seems to work fine with StGIT,
Well, it "works", but not "fine". Since it doesn't know about the stgit
objects, it won't ever pack them.
But maybe that's what stgit wants (since they are "temporary"), but it
does mean that if you see a big advantage from packing, you might be
losing some of it.
The 'git repack -a' command would include the applied patches in the
newly created pack but leave out the unapplied ones. It would be even
better to leave all of them out since the StGIT patches are frequently
changed but an independent mechanism for this would complicate GIT -
'git repack' shouldn't pack any of the objects found in
.git/refs/patches/, even if they are reachable via .git/refs/heads/*
(and maybe call the patches directory something like
.git/refs/unpackable or volatile).
--
Catalin
The preferred way would be to just list the references somewhere under
.git/refs/stgit, in which case fsck and repack should pick them up
automatically (so clearly stgit doesn't do that right now ;).
I thought about adding .git/refs/patches/<branch>/* files
corresponding to the every StGIT patch. Are the above git commands
looking at all depths in the .git/refs/ directory?
Yes. Or at least they're supposed to. If they are not, it's a bug
regardless, and we'll fix it.
The 'git repack -a' command would include the applied patches in the
newly created pack but leave out the unapplied ones. It would be even
better to leave all of them out since the StGIT patches are frequently
changed but an independent mechanism for this would complicate GIT -
'git repack' shouldn't pack any of the objects found in
.git/refs/patches/, even if they are reachable via .git/refs/heads/*
(and maybe call the patches directory something like
.git/refs/unpackable or volatile).
If we have some default location (and .git/refs/patches/ sounds good), we
can make git do the right thing - find them for git-fsck-objects, and
ignore them for git-repack.
Linus
From: Carl Baldwin <hidden> Date: 2016-06-15 22:42:13
On Mon, Nov 21, 2005 at 11:24:11AM -0800, Linus Torvalds wrote:
NOTE! Since that email, "git repack" has gotten a "local" option (-l),
which is very useful if the repositories have pointers to alternates.
So do
git repack -l
instead, to get much better packs (and "-a -d" for the full case, of
course).
I'm assuming that this option will have no effect on a repository with
no alternates file.
Other that than, the old email suggestion should still be fine.
[snip]
You can certainly do that if you are nervous. It might even be a good
idea: just for fun, I just did
git clone -l git git-clone
cd git-clone
# pick an object at random
rm .git/objects/f7/c3d39fe3db6da3a307da385a7a1cb563ed15f7
git repack -a -d
and it said:
error: Could not read f7c3d39fe3db6da3a307da385a7a1cb563ed15f7
fatal: bad tree object f7c3d39fe3db6da3a307da385a7a1cb563ed15f7
but then it created the pack _anyway_, and said:
Packing 27 objects
Pack pack-13bfca704078175c1c1c59964553b14f7b952651 created.
and happily removed all the old ones.
So right now, repacking a broken archive can actually break it even more.
Interesting.
NOTE! Your "git verify-pack" wouldn't even catch this: the _pack_ is fine,
it's just incomplete.
In my opinion, git repack did the right thing in creating the pack even
if it is more broken. Starting with a broken repository was the real
problem. git repack shouldn't need to worry too much about it.
Looking at it from the nervous repository admin's point of view I think
he would want to make sure that the repository is good to begin with. I
think this should be left up to the repository owner and maybe not git
repack. Although, the check that you do following this is probably a
good idea.
Of course, this only happens if the repository was broken to begin with,
so arguably it's not that bad. But it does show that git-repack should be
more careful and return an error more aggressively.
Can anybody tell me how to do that sanely? Right now we do
..
name=$(git-rev-list --objects $rev_list $(git-rev-parse $rev_parse) |
git-pack-objects --non-empty $pack_objects .tmp-pack) ||
exit 1
..
and the thing is, the "git-pack-objects" thing is happy, it's the
"git-rev-list" that fails. So because the last command in the pipeline
returns ok, we think it all is ok..
(This is one of the reasons I much prefer working in C over working in
shell: it may be twenty times more lines, but when you have a problem, the
fix is always obvious..)
Anyway, with that fixed, a "git repack" in many ways would be a mini-fsck,
so it should be very safe in general. Modulo any other bugs like the
above.
Linus
*NOTE* There is one question that I feel remains unanswered. Is it
possible to split up the repack -a and repack -d so that the nervous
repository owner can insert a git verify-pack in the middle.
I'm not nearly this nervous about repositories that I keep for myself
but I have ownership of some repositories on which many people may
depend. I will feel better if I can verify the pack separately from
git-repack before I do the (potentially destructive) -d to remove old
packs.
I don't mean to say that I don't trust git repack to do the right thing.
Fundamentally, I just think that I shouldn't depend on it to do the
right thing in order to avoid corruption in my repository.
Carl
PS I love that the git object store is designed so that object files
never *need* to be removed, renamed, modified or otherwise touched in
any way after being written to disk. I think this makes git inherently
extremely safe from corruption unlike many other older repository
designs. The only thing that breaks this inherent safety is the desire
to pack repositories to avoid bloat.
That is why I want to be a little paranoid when I do the repacking. I
want to maintain some inherent safety in the process that I use to pack
them. This kind of inherent safety is much more valuable then even the
highest quality code written to actually do the packing.
--
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Carl Baldwin Systems VLSI Laboratory
Hewlett Packard Company
MS 88 work: 970 898-1523
3404 E. Harmony Rd. work: Carl.N.Baldwin@hp.com
Fort Collins, CO 80525 home: Carl@ecBaldwin.net
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
On Mon, Nov 21, 2005 at 11:24:11AM -0800, Linus Torvalds wrote:
quoted
NOTE! Since that email, "git repack" has gotten a "local" option (-l),
which is very useful if the repositories have pointers to alternates.
So do
git repack -l
instead, to get much better packs (and "-a -d" for the full case, of
course).
I'm assuming that this option will have no effect on a repository with
no alternates file.
Correct.
The only thing it does is that when it looks up an object, if it's not in
our _own_ ".git/objects/" dir, it won't pack it.
Actually, that's not entirely true. It isn't smart enough to know where
every object exists, so it only knows about remote _packs_. So what
happens is that if you do
git repack -l -a -d
it will create a pack-file that contains _all_ unpacked objects (whether
local or not) and all objects that are in local packs (because of the
"-a"), but not any objects that are in "alternate packs".
Which is actually exactly what you want, if you are in the situation that
kernel.org is, and you have people who point their alternates to mine:
when I repack my objects, they'll use my packs, but other than that,
they'll prefer to use their own packs over any unpacked objects.
quoted
So right now, repacking a broken archive can actually break it even more.
Interesting.
Well, with the latest git repack script, that should no longer be true.
quoted
NOTE! Your "git verify-pack" wouldn't even catch this: the _pack_ is fine,
it's just incomplete.
In my opinion, git repack did the right thing in creating the pack even
if it is more broken. Starting with a broken repository was the real
problem. git repack shouldn't need to worry too much about it.
Well, "git repack" did the wrong thing in that it never _noticed_, and it
then removed all old packs - even though those old packs contained objects
that we hadn't repacked because of the broken repository.
Of course, _usually_ a broken repository is just that - broken. The way
you fix a broken repo is to find a non-broken one, and clone that.
However, sometimes what you can do (if you literally just lost a few
objects) is to find a non-broken repo, and make that the _alternates_, in
which case you may be able to save any work you had in the broken one
(assuming you only lost objects that were available somewhere else).
Looking at it from the nervous repository admin's point of view I think
he would want to make sure that the repository is good to begin with.
Doing an fsck is certainly always a good idea. I do a "shallow" fsck
usually several times a day ("shallow" means that it doesn't fsck packs,
only new objects that I have aquired since the last repacking), and I do a
full fsck a couple of times a week.
I don't actually know why I do that, though. I don't think I've really
_ever_ had a broken repo since some very early days, except for the cases
where I break things on purpose (like remove an object to check whether
"git repack" does the right thing or not). I'm just used to it, and the
shallow fsck takes a fraction of a second, so I tend to do it after each
pull.
So I really think that an admin has to be more than "nervous" to worry
about it. He has to be really anal.
(Now, doing a repack and a fsck every week or so might be good, and
automatic shallow fsck's daily is probably a great idea too. After all, it
_is_ checking checksums, so if you worry about security and want to make
sure that nobody is trying to break in and do bad things to your repo, a
regular fsck is a good thing even if you're not otherwise worried about
corruption).
*NOTE* There is one question that I feel remains unanswered. Is it
possible to split up the repack -a and repack -d so that the nervous
repository owner can insert a git verify-pack in the middle.
They are already split up inside "git-repack", so we could add a hook
there, I guess. See the git-repack.sh file, and notice how it does the
"remove_redundant" part only after it has created the new pack-file and
done a "sync".
I don't mean to say that I don't trust git repack to do the right thing.
Fundamentally, I just think that I shouldn't depend on it to do the
right thing in order to avoid corruption in my repository.
That's good. However, as the previous failure of git repack showed, to
some degree the more likely failure mode is actually that the pack
generated by "git repack" is perfectly fine, but it's not _complete_. Say
we have a bug in git repack, for example.
Another case where it's not complete is when you have deleted a branch.
"git repack -a -d" will effectively do a "git prune" wrt objects that are
no longer reachable, and that were in the old packs.
So I'd actually suggest a slightly different approach. When-ever you
remove old objects (whether it's "git prune" or "git prune-packed" or "git
repack -a -d"), you might want to have an option that doesn't actually
_remove_ them, but just moves them into ".git/attic" or something like
that.
Then you can clean up the attic after doing your weekly full fsck or
something. And it has the advantage that if somebody has deleted a branch,
and notices later that maybe he wanted that branch back, you can "unprune"
all the objects, run "git-fsck-objects --full" to find any dangling
commits, and you'll have all your branches back.
So in many ways it would perhaps be nicer to have that kind of "safe
remove" option to the pruning commands?
Linus
From: Chuck Lever <hidden> Date: 2016-06-15 22:42:13
Linus Torvalds wrote:
On Tue, 22 Nov 2005, Chuck Lever wrote:
quoted
there are some things repacking does that breaks StGIT, though.
git repack -d
seems to remove old commits that StGIT was still depending on.
If that is true, then "git-fsck-cache" probably also reports errors on a
StGIT repository. No? Basically, it implies that the tool doesn't know how
to find all the "heads".
indeed. this is one area where StGIT is "not safe" to use with other
porcelains. these raw GIT commands can show a bunch of confusing
"dangling references" type errors, or actually modify the index in ways
that eliminate StGIT-related commits that aren't currently attached to
any ancestry. (i think Catalin mentioned these are related to the
unapplied patches in a stack, but there could be others; see below).
The preferred way would be to just list the references somewhere under
.git/refs/stgit, in which case fsck and repack should pick them up
automatically (so clearly stgit doesn't do that right now ;).
that could be an extremely large number of commits on a large repository
with a lot of patches that have been worked on over a long period. so
whatever mechanism is created to do this needs to scale well in the
number of commits.
It also implies that doing a "git prune" will do horribly bad things to a
stgit repo, since it would remove all the objects that it thinks aren't
reachable..
yup. been there, done that. lucky for me i have an excellent hourly
backup scheme.
quoted
git repack -a -n
seems to work fine with StGIT,
Well, it "works", but not "fine". Since it doesn't know about the stgit
objects, it won't ever pack them.
ah!
But maybe that's what stgit wants (since they are "temporary"), but it
does mean that if you see a big advantage from packing, you might be
losing some of it.
actually, those commits aren't all that "temporary". the
history/revision feature i'm working on would like to maintain all the
commits ever done to an StGIT patch.
the only time you can throw away such commits is when the patch is
deleted or when it is finally committed to the repository via "stg
commit". otherwise, keeping these commits in a pack would be quite a
good thing.
maybe the first thing to do is to get a basic understanding of an StGIT
commit's lifetime.
But maybe that's what stgit wants (since they are "temporary"), but it
does mean that if you see a big advantage from packing, you might be
losing some of it.
actually, those commits aren't all that "temporary". the
history/revision feature i'm working on would like to maintain all the
commits ever done to an StGIT patch.
That's to avoid pruning them but you might not always want to add them
to a pack.
the only time you can throw away such commits is when the patch is
deleted or when it is finally committed to the repository via "stg
commit". otherwise, keeping these commits in a pack would be quite a
good thing.
maybe the first thing to do is to get a basic understanding of an StGIT
commit's lifetime.
My initial idea was to throw the old commit away once a patch is
refreshed. Even if you want to preserve the history, it would be only
preserved until you send the patch to be merged upstream and you would
delete it locally. If all the patches are meant to be sent upstream at
some point, you can avoid packing them.
--
Catalin