usage for git-p2p (was: git pack/unpack over bittorrent - works!)

On Sat, Sep 4, 2010 at 2:52 AM, Artur Skawina [off-list ref] wrote:

Hmm, taking a few steps back, what is the expected usage of git-p2p?
Note it's a bit of a trick question; what i'm really asking is what _else_,
other than pulling/tracking Linus' kernel tree will/can be done with it?

 i'm _so_ glad you asked :)  please note - for all of these i'm keenly
aware that GPG signing is required (and nicolas has pointed out that
you only need a 20-byte hash to be securely transferred by some OOB
mechanism)

 * distribution of large mailing lists and reduction of load on SMTP
servers by adding in a firebreak layer where the SMTP data goes into a
git repository first, then is extracted on the other side.  users may
choose to download the mailing list using git over a peer-to-peer
network and then use the exact same software that the mailing list
server is running, thus reducing the load on the web front-end.  if
you then choose also to allow users to GPG-sign commits to the same
git repository, you have the possibility of reducing the load on the
SMTP servers *as well*.  _and_ a group of users can operate in
"offline" mode whilst still being able to collaborate (by sharing the
git repo between themselves on an isolated LAN) _and_ when one of them
goes back "online", all their commits sync up, and the rest of the
world becomes aware of that group's offline discussion.

 * distributed wikis.  there are lots of wikis now using git: ikiwiki
is one, and moinmoin can also be configured to use git.  so, wikis
have already solved the "conflict" problem, in particular i know that
joey is a smart bunny and has solved conflict resolution in ikiwiki.
that makes it possible to simply turn an "ordinary" wiki into a
"peer-to-peer" wiki by merely inserting git-p2p underneath.

 * distributed bugtrackers.  the one that i know of which i believe
would work straight away is the ikiwiki "bug" plugin.  the rest i need
to investigate: i know of at least one that's based on git that i
haven't looked at for a loong while (ditrack) - but the principle is
very straightforward: create bugs by UUID, commit, distribute.  if you
want a "central" numbering scheme then all that's needed is for a
"central" service to allocate a "central" number - a symlink to the
UUID would do - and you're done.

 * of course there's source management for other projects other than
linux-2.6 :)

there's plenty more ideas out there, but the primary ones i'm
interested in are the ones where "traditional" software development
tools are being used which are utterly dependent on the client-server
paradigm.  for reasons i won't explain as it's not relevant to this
technical discussion list, this is making me feel a bit twitchy [free
software development being dependent on the client-server paradigm,
that is].

l.

On 09/04/10 13:54, Luke Kenneth Casson Leighton wrote:

On Sat, Sep 4, 2010 at 2:52 AM, Artur Skawina [off-list ref] wrote:

quoted

Hmm, taking a few steps back, what is the expected usage of git-p2p?
Note it's a bit of a trick question; what i'm really asking is what _else_,
other than pulling/tracking Linus' kernel tree will/can be done with it?

 i'm _so_ glad you asked :)  please note - for all of these i'm keenly
aware that GPG signing is required (and nicolas has pointed out that
you only need a 20-byte hash to be securely transferred by some OOB
mechanism)

 * distribution of large mailing lists and reduction of load on SMTP

 * distributed wikis.  there are lots of wikis now using git: ikiwiki

 * distributed bugtrackers.  the one that i know of which i believe

 * of course there's source management for other projects other than
linux-2.6 :)

I'm not sure that git fits well any of the above; it may be convenient
to reuse git as it already exists and can be more or less easily plugged
in, but it's probably not the best backend choice. The one that might
map relatively closely to git is wikis, but there i'm not sure something
more scalable wouldn't be better (eg how would wikipedia map onto git..)


The one other use case for git-p2p that i can see is distributing the
object store over a LAN. But it's really about a _remote_ object store,
not so much a distributed one.
IOW, having git-checkout trying to reach "/tmp/.git-unix/G0" and then any
remote object store in addition to ".git/objects" would be very useful.
But, do you actually want to spread the objects on _every_ developer
desktop on the LAN? Wouldn't it much more make sense to have two or three
dedicated "object servers", which would give you enough redundancy and,
because of the higher chances of finding the required objects in cache,
also perform better?

For the case of an ad-hoc group of developers, the protocol described in
my previous email would work well, they only need to find one peer and
then can move objects back and forth in the swarm.


One of the advantages that git-p2p could bring is better packing -- instead
of everyone running git-repack with a limited window, each peer in the swarm
could better use its resources by looking much deeper for the perfect delta
candidate, but only for a subset of the objects, and then share the results
with the rest. As a "git-repack -a -f" can eg shrink the gcc.git repo by ~15%
(~100M) the savings could be noticeable.

artur

On Sat, Sep 4, 2010 at 4:08 PM, Artur Skawina [off-list ref] wrote:

The one other use case for git-p2p that i can see is distributing the
object store over a LAN.

 ok - then you should help with the "git hive" effort that casey's
writing, as he is focussing on writing an extension to git which does
exactly that.  the discussion on this thread is focussed on extending
git to work out on the wider internet, not just the case of optimised
transfers on a LAN segment.

l.

Luke Kenneth Casson Leighton wrote:

ikiwiki is one, and moinmoin can also be configured to use git. Wikis
have already solved the "conflict" problem, in particular i know that
joey is a smart bunny and has solved conflict resolution in ikiwiki.

More accurately, I have entirely punted conflict resolution onto the
underlying VCS, such as git, and taken advantage of its generally
excellent merging behavior to avoid the most common classes of things
that result in conflicts/locking on other wikis (like two users editing
different parts of the same page concurrently).

Two branches of a wiki with committers to both can still of course
result in conflicts. In this case, it might make sense to avoid conflict
resolution during merging, and make the conflicted page be committed
containing both versions and some sort of "conflict here!" warning
directive. Then the wiki software can provide an web interface for
cleaning up the conflict later. A custom merge driver could do something
like that, although there's the problem that not all files in the wiki
will be wiki pages, or the conflict could involve conflicting renames of
pages. (Another approach would be to use the "ours" merge strategy.)

-- 
see shy jo, lazy bunny