Re: backup or mirror a repository

Dan Farina [off-list ref] writes:

I did look at prune and update, but my problem is the opposite: I want
something that will remove branches from the remote repo when they no
longer exist locally. As-is over time I will proliferate little local
branches unless I occasionally sit down and delete branches by operating
directly on the bare backup repository. (and then use prune on the
remote nodes)

The "git remote add --mirror" setup is about setting up the
local repository _AS_ the backup of the remote.  In other words,
the contents come from the remote by fetching from it and safely
kept away from disaster on the local side.  And for that,
"remote prune" is a perfect thing to do.

I think what you are asking for is an opposite, a backup remote
site you would push into.  That is not what "remote add --mirror"
is about.

You can almost do it with

	git push --all $remote

except there is no way to automagically remove the branch you
removed from the local repository.  For that, we would need a
new --mirror option to "git-push".

I think it is trivial to do for native transports, as we first
get the list of all refs from the remote side before starting
the transfer.  You need to change the last parameter called
'all' to remote.c::match_refs() into an enum ('push_all' being
one of choices), introduce another enum 'push_mirror', and teach
it to "match" the remote (i.e. dst) ref that does not have
corresponding entry on our side (i.e. src) with an empty object
name to mark it removed.  Then the part marked as "Finally, tell
the other end!"  in send-pack.c::send_pack() will take care of
the actual removal.

On Thu, 2007-09-27 at 23:27 -0700, Junio C Hamano wrote:

You can almost do it with

	git push --all $remote

except there is no way to automagically remove the branch you
removed from the local repository.  For that, we would need a
new --mirror option to "git-push".

Yup! I actually was looking for such a thing in the git-push man page,
but was unsuccessful. I was living with git-push --all for a little
while before I thought I'd ask. Unfortunately, the preferred setup
(having the backup machine actively perform fetch and prune) is not very
nice for me due to firewalls.

I think it is trivial to do for native transports, as we first
get the list of all refs from the remote side before starting
the transfer.  You need to change the last parameter called
'all' to remote.c::match_refs() into an enum ('push_all' being
one of choices), introduce another enum 'push_mirror', and teach
it to "match" the remote (i.e. dst) ref that does not have
corresponding entry on our side (i.e. src) with an empty object
name to mark it removed.  Then the part marked as "Finally, tell
the other end!"  in send-pack.c::send_pack() will take care of
the actual removal.

I should have a look, but if someone else wants to work on this they
shouldn't block on me: work already made me get hit by a bus on one
project (luckily I wasn't important), and the storm is not over.

Thanks,
fdr

On Thu, Sep 27, 2007 at 11:27:06PM -0700, Junio C Hamano wrote:

The "git remote add --mirror" setup is about setting up the
local repository _AS_ the backup of the remote.  In other words,
the contents come from the remote by fetching from it and safely
kept away from disaster on the local side.  And for that,
"remote prune" is a perfect thing to do.

I have tried to do that but I am getting a warning:
$ git remote prune origin
Warning: unrecognized mapping in remotes.origin.fetch: +refs/*:refs/*
and no branch is removed.

I suspect that the change that introduced --mirror option for the 'add'
command did not adjust the prune procedure to handle the new situation
properly. Or is just me doing something wrong?

Anyway, because the official released git still does not have --mirror
and it could be difficult to convince the admin to use the bleading edge
version of Git for secure backup, I wrote a simple script instead, which
provides the same functionality but using the released version of Git.
Maybe, someone else finds it useful.

$ cat git-mirror
#!/bin/sh

usage() {
	echo >&2 "Usage: $0 URL"
	exit 1
}

set -e

test $# -eq 1 || usage
URL="$1"

# Initialize Git bare directory
git --bare init

# Initialize the mirror
git remote add origin "$URL"
git fetch
rmdir refs/heads
ln -s remotes/origin refs/heads

# Adding creating a script that will be run by cron
cat <<EOF > git-mirror-sync
cd "$PWD"
git fetch
git remote prune origin
EOF
chmod +x git-mirror-sync

# Adding git-mirror-sync to cron
echo "Please, add $PWD/git-mirror-sync to cron"

#########################################################

Dmitry

Hi,

On Thu, 18 Oct 2007, Dmitry Potapov wrote:

On Thu, Sep 27, 2007 at 11:27:06PM -0700, Junio C Hamano wrote:

quoted

The "git remote add --mirror" setup is about setting up the
local repository _AS_ the backup of the remote.  In other words,
the contents come from the remote by fetching from it and safely
kept away from disaster on the local side.  And for that,
"remote prune" is a perfect thing to do.

I have tried to do that but I am getting a warning:
$ git remote prune origin
Warning: unrecognized mapping in remotes.origin.fetch: +refs/*:refs/*
and no branch is removed.

I suspect that the change that introduced --mirror option for the 'add' 
command did not adjust the prune procedure to handle the new situation 
properly. Or is just me doing something wrong?

No, you're right.  I did not anticipate git-remote to be written the way 
it is.  After fiddling with it for several hours, I am giving up for now.  
IMHO the script is too married to the idea that the remote branches live 
in refs/remotes/<remote>/*.  Probably it would be very easy by now to 
implement it as a builtin, using remote.[ch].

Ciao,
Dscho