Linus Torvalds [off-list ref] writes:
We really should _never_ create a pack in-place with the final name.
The "fattening" index-pack Nico did does not have this problem
as far as I can see. Under --stdin, it creates a temporary pack
file "$GIT_OBJECT_DIRECTORY/pack_XXXXXX"; after the received
pack is fattened by adding missing delta-base objects and fixing
up the pack header, final() moves the {pack,idx} pair to the
final location.
The race is about this sequence:
- git-receive-pack is spawned from remove git-send-pack;
it lets "index-pack --stdin --fatten" to keep the pack.
- index-pack does its magic and moves the pack and idx
to their final location;
- "repack -a -d" is started by somebody else; it first
remembers all the existing packs; it does the usual
repacking-into-one.
- git-receive-pack that invoked the index-pack waits for
index-pack to finish, and then updates the refs;
- "repack -a -d" is done repacking; removes the packs
that existed when it checked earlier.
Two instances of receive-pack running simultaneously is safe (in
the sense that it does not corrupt the repository; one instance
can fail after noticing the other updated the ref it wanted to
update) and there is no reason to exclude each other. But
"repack -a -d" and receive-pack are not.
Can we perhaps have reader-writer lock on the filesystem to
pretect the repository? "prune" can also be made into a writer
for that lock and "fetch-pack --keep" would be a reader for the
lock. That reader-writer lock would solve the issue rather
nicely.
That said, I think some of the "git repack -d" logic is also unnecessarily
fragile.
Noted; will fix.
On Fri, 27 Oct 2006, Shawn Pearce wrote:
So a reader-writer lock is preferred over
a non-locking solution such as I posted in
http://article.gmane.org/gmane.comp.version-control.git/30288 ?
Not to mention that such a solution would also fix the -d issue
Linus points out above.
Be very careful.
There's a good reason why git doesn't use locking, and tends to use the
"create file exclusively and move over the old version after having tested
that the old version is still relevant" approach.
Two _major_ issues:
- just about any other locking algorithm simply doesn't work on some
filesystems. And then you're just royally screwed.
- I want to be able to push out, regardless of whether there is somebody
(or millions of somebodies) reading the repository at the same time. So
locking is not acceptable for "normal operations" at all - at most this
would be a "keep a repack from interfering with another repack" kind of
thing.
I would MUCH rather we just rename the index/pack file to something that
git can _use_, but that "git repack -a -d" won't remove. In other words,
rather than locking, it would be much better to just use a naming rule:
when we download a new pack, the new pack will be called
new-pack-<SHA1ofobjectlist>.pack
new-pack-<SHA1ofobjectlist>.idx
and we just make the rule that "git repack -a -d" will only ever touch
packs that are called just "pack-*.{pack|idx}", and never anything else.
It really is that simple. Allow normal git object opens to open the
"temporary file" naming version too (so that you can install the refs
before the rename, and all the objects will be visible), but don't allow
"git repack" to remove packs that are in the process of being installed.
Race removed, and no locking really needed. At most, we might need to be
able to match up a "new-pack-*.idx" file with a "pack-*.pack" file when we
open pack-files, simply because we can't rename two files atomically, so
the pack-file and index file would potentially exist with "different"
names for a short window.
That kind of small semantic changes are _way_ better than introducing
locking, which will inevitably have much worse error cases (not working,
stale locks, inability to push because something is really slow, or any
number of other problems).
Junio C Hamano [off-list ref] wrote:
Linus Torvalds [off-list ref] writes:
[snip]
Can we perhaps have reader-writer lock on the filesystem to
pretect the repository? "prune" can also be made into a writer
for that lock and "fetch-pack --keep" would be a reader for the
lock. That reader-writer lock would solve the issue rather
nicely.
quoted
That said, I think some of the "git repack -d" logic is also unnecessarily
fragile.
Noted; will fix.
So a reader-writer lock is preferred over
a non-locking solution such as I posted in
http://article.gmane.org/gmane.comp.version-control.git/30288 ?
Not to mention that such a solution would also fix the -d issue
Linus points out above.
--