Re: Alternates and push

Theodore Tso [off-list ref] writes:

Long-standing mis-feature in git's logic in deciding what to push.
It's been reported a few times, but apparently it's hard to fix, or at
least it never hsa been fixed as far as I know.

This comes from an early (mis)design of git.

Background.

 * A git repository and the object store it uses can be separate.  From
   the beginning, you can have a"objects/" directory (aka "object store")
   that is shared by more than one repositories.  There is no Porcelain
   level support to set up two repositories that physically share the same
   object store, but the result of "git init; rm -rf .git/objects; ln -s
   $other/.git/objects .git/objects" was supposed to work (and it still
   largely works, until you gc) in the original design.

   The alternate object store does not even have to be a git repository,
   which makes things worse.  You can have everybody pointing at
   /var/cache/objects, and /var/cache does not have to be a git repository
   (i.e. no var/cache/refs).

 * The existing alternates mechanism is not about alternate repositories.
   It is about alternate object stores.  That is why each line of this
   file points at "objects" directory elsewhere, not the ".git" directory
   that is typically at one level above that "objects" directory.

   The fact your repository's object store points at the object store that
   happens to be inside Linus's repository does not imply that Linus's
   object store is associated with refs in Linus's repository in any way
   (that's the early _mis_design part).

 * An existing ref in a git repository is meant to be a guarantee that all
   objects the object referenced by the ref is found somewhere in the
   object store(s) the repository uses.  Object transfers in git
   (i.e. fetch and push) use this guarantee to tell what a repository has
   to the other side.

   What happens in your case is that github end knows that the repository
   you are pushing into have up to the refs you have there.  Alternate may
   point at object store that holds objects from Linus's repository, but
   there is no information as to what the latest commits you do not see in
   your refs namespace (namely, "what's Linus's latest" is not something
   you can learn from your repository that has alternates).

A possible fix would involve:

 - Deprecate objects/info/alternates file, and GIT_OBJECT_DIRECTORY and
   GIT_ALTERNATE_OBJECT_DIRECTORIES environment variables;

 - Introduce info/alternates that points at alternate _repositories_ (as
   opposed to objects/info/alternates that points at alternate object
   stores);

 - Teach fetch and push to include refs from alternate _repositories_ into
   what local side considers complete.

The above won't break existing setups, but it won't help them either.  All
the borrowing repositoies need to be converted if we go that route.

We could instead redefine the semantics of the existing alternates
mechanism.  This technically *breaks* backward compatibility, but I
suspect it won't hurt many existing installations:

 - Declare that a freestanding object store is illegal.  In other words,
   if a directory "$D/objects" is (1) used as $GIT_OBJECT_DIRECTORY's
   value, (2) pointed by some repository's "alternates" file, or (3)
   listed in $GIT_ALTERNATE_OBJECT_DIRECTORIES's value, this change makes
   it illegal for "$D" not being a proper git repository.

   This will not break your example of your repository's object store
   borrowing from the object store inside Linus's repository.

 - When you have "$D/objects" in alternates, start relying on "$D/refs"
   being correct (i.e. repository $D is not corrupt).  This technically
   makes the system slightly less robust, as we are depending on _other
   people's_ good behaviour even more when you use alternates, but you are
   already depending on them having good objects in $D/objects anyway, so
   it is not a big deal.

 - Now that we declared that everything reachable from "$D/refs" do not
   have to be transferred from elsewhere when a push sends things into us
   (or a fetch gets things from elsewhere into us) when you have
   "$D/objects" in your alternates.  In your "borrowing from Linus"
   example, Linus's latest will be reachable from somewhere in "$D/refs",
   when you are borrowing from him by having "$D/objects" in your
   alternates. 

On 9/6/08, Junio C Hamano [off-list ref] wrote:

Theodore Tso [off-list ref] writes:

 > Long-standing mis-feature in git's logic in deciding what to push.
 > It's been reported a few times, but apparently it's hard to fix, or at
 > least it never hsa been fixed as far as I know.


This comes from an early (mis)design of git.

What happens if I set a remote in my github digispeaker repo pointing
to github's linus repo and the set a chron script to fetch it once a
day? The alternates link is still in place.

-- 
Jon Smirl
jonsmirl@gmail.com

Jon Smirl [off-list ref] wrote:

On 9/6/08, Junio C Hamano [off-list ref] wrote:

quoted

Theodore Tso [off-list ref] writes:

 > Long-standing mis-feature in git's logic in deciding what to push.
 > It's been reported a few times, but apparently it's hard to fix, or at
 > least it never hsa been fixed as far as I know.


This comes from an early (mis)design of git.

What happens if I set a remote in my github digispeaker repo pointing
to github's linus repo and the set a chron script to fetch it once a
day? The alternates link is still in place.

github should do what repo.or.cz does:

	ln -s .../linus.git/refs digispeaker.git/refs/forkee

That way the refs available in Linus' tree are also available
in your tree as Git will transparently follow the symlink.

However you have to be careful to make sure `git pack-refs` isn't
run with `--all --prune` as it will delete the refs from linus.git
when executed in digispeaker.git.  Fun times when I did that to my
own repository one day.  ;-)

Though we probably should fix Git to be somewhat smarter.  But
until that stable binary is available, the symlink trick above
is a good work around.

-- 
Shawn.

On Sat, Sep 06, 2008 at 11:06:49 -0700, Junio C Hamano wrote:

Theodore Tso [off-list ref] writes:

quoted

Long-standing mis-feature in git's logic in deciding what to push.
It's been reported a few times, but apparently it's hard to fix, or at
least it never hsa been fixed as far as I know.

This comes from an early (mis)design of git.
[...]
 * The existing alternates mechanism is not about alternate repositories.
   It is about alternate object stores.  That is why each line of this
   file points at "objects" directory elsewhere, not the ".git" directory
   that is typically at one level above that "objects" directory.

   The fact your repository's object store points at the object store that
   happens to be inside Linus's repository does not imply that Linus's
   object store is associated with refs in Linus's repository in any way
   (that's the early _mis_design part).

Why is this a *mis*design? Couldn't push be fixed by redesigning it's
protocol along the lines of:
 - clients sends a list of sha1s it wants to push, from the tip down
 - server stops it when it sees an object it has -- this check can be done
   against the object store without having a ref for it.

Regards,
Jan

-- 
						 Jan 'Bulb' Hudec [off-list ref]

On Sat, Sep 06, 2008 at 11:06:49AM -0700, Junio C Hamano wrote:

We could instead redefine the semantics of the existing alternates
mechanism.  This technically *breaks* backward compatibility, but I
suspect it won't hurt many existing installations:

 - Declare that a freestanding object store is illegal.  In other words,
   if a directory "$D/objects" is (1) used as $GIT_OBJECT_DIRECTORY's
   value, (2) pointed by some repository's "alternates" file, or (3)
   listed in $GIT_ALTERNATE_OBJECT_DIRECTORIES's value, this change makes
   it illegal for "$D" not being a proper git repository.

   This will not break your example of your repository's object store
   borrowing from the object store inside Linus's repository.

 - When you have "$D/objects" in alternates, start relying on "$D/refs"
   being correct (i.e. repository $D is not corrupt).  This technically
   makes the system slightly less robust, as we are depending on _other
   people's_ good behaviour even more when you use alternates, but you are
   already depending on them having good objects in $D/objects anyway, so
   it is not a big deal.

One way that wouldn't break backwards compatibility would be to use
$D/refs if it exists, but if it isn't, fall back to existing behavior
(which is to say, only use the refs in the repository, not in the
borrowed repository/object store).  Is there a reason why this would
be problematic?

							- Ted

On Sat, Sep 06, 2008 at 12:21:06PM -0700, Shawn O. Pearce wrote:

github should do what repo.or.cz does:

	ln -s .../linus.git/refs digispeaker.git/refs/forkee

That way the refs available in Linus' tree are also available
in your tree as Git will transparently follow the symlink.

However you have to be careful to make sure `git pack-refs` isn't
run with `--all --prune` as it will delete the refs from linus.git
when executed in digispeaker.git.  Fun times when I did that to my
own repository one day.  ;-)

Though we probably should fix Git to be somewhat smarter.  But
until that stable binary is available, the symlink trick above
is a good work around.

But it should be used only in controlled environment. If you happen to
have permissions to write to the forkee, you can wipe out its refs with
git fetch --mirror (and if you don't happen to have permissions, it will
just fail, so currently you cannot use this on repo.or.cz forks,
unfortunately). If you don't make sure refs are never packed (I do on
repo.or.cz, historically because of dumb transports - do they support
packed refs by now?), this won't work either. Maybe there are other
considerations too.

-- 
				Petr "Pasky" Baudis
The next generation of interesting software will be done
on the Macintosh, not the IBM PC.  -- Bill Gates

Petr Baudis [off-list ref] wrote:

On Sat, Sep 06, 2008 at 12:21:06PM -0700, Shawn O. Pearce wrote:

quoted

github should do what repo.or.cz does:

	ln -s .../linus.git/refs digispeaker.git/refs/forkee

But it should be used only in controlled environment. If you happen to
have permissions to write to the forkee, you can wipe out its refs with
git fetch --mirror (and if you don't happen to have permissions, it will
just fail, so currently you cannot use this on repo.or.cz forks,
unfortunately). If you don't make sure refs are never packed (I do on
repo.or.cz, historically because of dumb transports - do they support
packed refs by now?), this won't work either. Maybe there are other
considerations too.

All very good points.

The dumb transports do support packed-refs in more modern versions.
I forget when they learned it though.  Being nice to them by not
packing refs is still good if you can.

-- 
Shawn.