Re: Git-aware HTTP transport

On Mon, Sep 1, 2008 at 11:06 PM, Shawn O. Pearce [off-list ref] wrote:

quoted

What does the server do after a 000cgive-up ?  Does the server send
back a complete pack (like a new clone) or if not, how does clone work
over smart http?

When the server receives a "give-up" it needs to create a pack
based on "git rev-list --objects-boundary $WANT --not $COMMON".
If the set $COMMON is non-empty then its a partial pack; if $COMMON
is empty then its a full clone.  This is what the native protocol
does when the client gives up.

OK, that makes sense now.

quoted

Does that mean that if I fall more than 256 commits
behind, I have to redownload the whole repo?

You are thinking the wrong way.  If you have more than 256 commits
that the other side doesn't have you may give up too early.
For that to be true you need to create 256 commits locally that
aren't on the remote peer and whose timestamps are all ahead of
the commits you last fetched from the remote peer.

Yes, it can happen.  But its less likely than you think because
we're talking about you doing 256 commits worth of development and
not picking up any new commits from remote peers in the middle of
that time period.  Get just one and it resets the counter back to
0 and allows it to try another 256 commits before giving up.

I should amend this section to talk about what giving up here
really means.  If we have nothing sent in common yet or maybe
very little sent in common we may have existing remote refs tied
to this URL in .git/config that can send, and we may have one or
more annotated tags that we know for a fact are in common as both
peers have the same tag name pointing to the same tag object.

A smart(er) client might try to toss some recently dated annotated
tags at the server before throwing a give-up if it would otherwise
throw a give-up.  Its likely to narrow the result set, and doesn't
hurt if it doesn't.

Yes, throwing in tags and remotes as a last resort sounds like a good idea.

quoted

So again, if the client falls more than 1000 commits behind (not hard
to do for example during the linux merge window), and then the client
WANTs HEAD^1001, what happens?  Does the get nothing from the server,
or does the client essentially reclone, or I am missing something?

Oh, this is a live-lock condition.  If the client grabs the list of
refs from the server, then has to wait 100 ms to get back to the
server and start upload-pack (due to latency) and in that 100ms
window Linus shoves a 1001 commit merge into his tree then yes,
the server may abort and tell the client "error invalid want".

Ahh, now I get it.  Somehow I forgot that the WANTs were only boundary
commits and not a list of all the commits that the client wants.

On Mon, Sep 1, 2008 at 11:13 PM, Shawn O. Pearce [off-list ref] wrote:

"H. Peter Anvin" [off-list ref] wrote:

quoted

Shawn O. Pearce wrote:

quoted

Correct.  Today _none_ of the transport protocols allow the server
to force the client to use some sort of reference repository for an
initial clone.  There are likely two reasons for this:

 *) Its a lot simpler to program to just get everything from
    one location.

 *) If you really are forking an open source project then in
    some cases you may need to distribute the full source,
     not your delta.  You may just as well distribute the full
     source and call it a day.

3) it encourages single points of failure.

Or bad network usage, as I pointed out later about an India user
unknowingly being forced into a US based mirror when another was
closer to them.

I didn't make it clear in my response but I'm really against our
protocol having this sort of explicit redirect.  I'd rather put a
requirement in that says "Unless you have X,Y,Z in common with me
(directly or indirectly) I'm just not going to give you a pack".

OK, this all makes sense. http:// and git:// are probably the wrong
protocols to reduce bandwidth for the server for new clones.  Long
term, maybe gittorrent will be the right solution...

Thanks,
Tarmigan