Re: git pack/unpack over bittorrent - works!

Nicolas Pitre [off-list ref] writes:

quoted

* enumerate the list of objects associated with an individual commit by:
    i) creating a CUSTOM pack+idx using git pack-objects {ref}
    ii) *parsing* the idx file using gitdb's FileIndex to get the list
of objects

That's where you're going so much out of your way to give you trouble.  
A simple rev-list would give you that list:

	git rev-list --objects <this_commit> --not <this_commit''s_parents>

That's it.

I didn't want to get into this discussion, but where in the above picture
does the usual "want/ack" exchange fit?

The biggest trouble before object transfer actually happens is that the
sending end needs to find a set of commits that are known to exist at the
receiving end, but it needs to do that starting from a state where the tip
commits the receiving end has are not known by it.  That is why the
receiver must go back in his history and keep asking the sender "I have
this, this, this, this...; now have you heard enough?"  until the sender
sees a commit that it knows about.  After that happens, it can list them
on its "git rev-list --objects <my tips> --not <he has these>" command
line in order to enumerate objects it needs to send.

If the receiver is purely following the sender, not doing any work on its
own, the tips the receiver has may always be known by the sender, but that
is not an interesting case at all.

On Mon, 6 Sep 2010, Junio C Hamano wrote:

Nicolas Pitre [off-list ref] writes:

quoted

quoted

* enumerate the list of objects associated with an individual commit by:
    i) creating a CUSTOM pack+idx using git pack-objects {ref}
    ii) *parsing* the idx file using gitdb's FileIndex to get the list
of objects

That's where you're going so much out of your way to give you trouble.  
A simple rev-list would give you that list:

	git rev-list --objects <this_commit> --not <this_commit''s_parents>

That's it.

I didn't want to get into this discussion, but where in the above picture
does the usual "want/ack" exchange fit?

Before object enumeration obviously.  But I think that Luke has enough 
to play with already by only assuming the easy case for now.  If Git P2P 
is to be viable, it has to prove itself at least with the easy case 
first.


Nicolas

On Tue, Sep 7, 2010 at 12:57 AM, Nicolas Pitre [off-list ref] wrote:

On Mon, 6 Sep 2010, Junio C Hamano wrote:

quoted

Nicolas Pitre [off-list ref] writes:

quoted

quoted

* enumerate the list of objects associated with an individual commit by:
    i) creating a CUSTOM pack+idx using git pack-objects {ref}
    ii) *parsing* the idx file using gitdb's FileIndex to get the list
of objects

That's where you're going so much out of your way to give you trouble.
A simple rev-list would give you that list:

    git rev-list --objects <this_commit> --not <this_commit''s_parents>

That's it.

I didn't want to get into this discussion, but where in the above picture
does the usual "want/ack" exchange fit?

Before object enumeration obviously.  But I think that Luke has enough
to play with already by only assuming the easy case for now.  If Git P2P
is to be viable, it has to prove itself at least with the easy case
first.

 :)   yes.  worry about that later.  optimisation.  time to think.
idea earlier (from 2 hours ago) unworkable, thought of another one,
split commit list into multi-level "virtual hierarchical
subdirectories" of say 256 entries each.  can therefore easily trip
down each "subdirectory" which will quickly get you to the right place
where the commits are different, with only a few roundtrips.  sort-of
binary search but 256-way search.  binary search not optimal here
because of multiple network round-trips.  sorry very obtuse will write
up better.

l.

On Tue, Sep 7, 2010 at 12:57 AM, Nicolas Pitre [off-list ref] wrote:

 But I think that Luke has enough
to play with already by only assuming the easy case for now.

 um, yes.

 i only just noticed that bittorrent's multi-file mode has blocking
that doesn't line up with the bloody file beginnings and ends:

| block 0 256k | block 1 256k | block N 11bytes |
|file1 | file2          | file3 | file4               |

this is why cameron dale threw his hands up in horror at bittorrent
when he created debtorrent, but i didn't understand why, fully, as i
thought that he was creating one .torrent per .deb _anyway_ so if he
had it would have been moot.

so i have to do some tests to see if multiple individual torrents can
be added using the BitTornado API to the same server, and redesign the
damn code so that it adds new "files" to be downloaded based on the
previous one completing.

so, the first quotes file quotes to be requested will be the
"rev-list", and the "finished" callback function will result in the
next layer of "files" as .torrents to be requested, and then finally
the files representing the actual "objects" - generated as git
cat-files on the remote end - get added and requested.

so - all event-driven.  eek!  fuun.  about as understandable as nmbd
in samba (*) :)

l.

(*) i can say that because i did its 1st major rewrite ha ha