git-bpush: Pushing to a bundle

Hi *,

  I've made a script to push to a bundle that tries to behave as the
normal push.

  It has some limitations, but for the normal cases it works fine.

The basic idea is:

- Easily create bundles with the current branch.
- Be able to push to defined bundles in remote.<remote>.url
- Only add new objects by default (do not lose objects)
- Reuse existing bundles (keep the basis and the branches)
- Check that the branches fast-forward
- Keep the pushed branches in refs/remotes/<remote>/*

But it also has some limitations:

- Do not allow refspec (git-bundle do not support them), only branch/tags names.
- Push all branches or none (consequence of the above)
- ...

Hope this helps,
Santi

The scripts follows, but also attatched.
#!/bin/sh

OPTIONS_KEEPDASHDASH=
OPTIONS_SPEC="\
git bpush [options] [<remote> [<refs>...]]
--
f,force   force updates
full      create a full bundle
v         be verbose
"
SUBDIRECTORY_OK=Yes
. git-sh-setup
. git-parse-remote

cd_to_toplevel

LF='
'
IFS="$LF"

bases=
bbases=
changed=
force=
nonff=
remote=
refs=
while [ $# != 0 ] ; do
    case "$1" in
	-v) verbose=t;;
	--full) full=t;;
	-f|--force) force=t;;
	--) shift; break;;
	*) usage;;
    esac
    shift
done

[ -n "$1" ] && remote=$1 && shift
while [ $# != 0 ] ; do
    refs="$refs$LF$1" && shift
done

[ -z "$remote" ] && remote=$(get_default_remote)
remoteurl=$(git config remote.${remote}.url)
[ -z "$remoteurl" ] && remoteurl=$remote
[ -d "$remoteurl" ] && die "$remoteurl is a directory"

# Default bases in bundle.base
# Default {refs,base} can be specified in remote.<remote>.{push,bundlebase}
if [ "$remote" != "$remoteurl" ] ; then
    [ -z "$refs" ] &&
    refs=$(git config --get-all remote.${remote}.push)
    bases=$(git config --get-all remote.${remote}.bundlebase ||
	git config --get-all bundle.base)
else
    bases=$(git config --get-all bundle.base)
fi

# git rev-parse --symbolic-full-name resolves symlinks
# Keep at least HEAD
head=
for ref in $refs ; do
    [ "$ref" = HEAD ] && head=t && break
done

[ -n "$bases" ] && bases=$(git rev-parse --revs-only $bases | sort -u)
# Full symbolic refs to be uniq
[ -n "$refs" ] && \
    refs=$(git-rev-parse --symbolic-full-name --revs-only $refs | sort -u) && \
    [ -n "$head" ] && refs="HEAD$LF$refs"

if [ -e "$remoteurl" ] ; then
    # Find the bundle's bases
    refs="$refs$LF$(git bundle list-heads $remoteurl | cut -d " " -f 2)"
    requires=
    for line in $(git bundle verify "$remoteurl" 2>/dev/null) ; do
	case "$line" in "The bundle requires"*) requires=t && continue; esac
	[ -z "$requires" ] && continue
	bbase=$(echo $line | cut -d " " -f 1)
	[ -z "$bbases" ] && bbases=$bbase && continue
	bbases="$bbases$LF$bbase"
    done
    bbases=$(echo "$bbases" | sort -u)
    [ -z "$bases" ] && bases="$bbases" || bases="$bases$LF$bbases"
elif [ -z "$refs" ] ; then
    # Push current branch
    refs="HEAD$LF$(git symbolic-ref -q HEAD)"
fi

[ -z "$refs" ] && die "No refs to push"

refs=$(echo "$refs" | sort -u)

for ref in $bases $refs ; do
    [ "$(git cat-file -t $ref^{})" != commit ] && \
	die "$(basename $0): $ref is not a commit"
done

header="To $remoteurl"
[ -n "$verbose" ] && echo "Pushing to $remoteurl" && echo $header && header=

# Find what is/is not a fast-forward, up to date or new
# As "git bundle" does not support refspecs we must push all matching branches
for ref in $refs ; do
    case $ref in
	refs/tags/*) type=tags; newtext="new tag";;
	refs/heads/*|HEAD) type=heads ; newtext="new branch" ;;
    esac
    newhash=$(git rev-parse $ref)
    newshort=$(git rev-parse --short $ref)
    bshort=$(echo $ref | sed -e "s|^refs/$type/||")
    if [ -e "$remoteurl" ] ; then
	bheads="$(git bundle list-heads $remoteurl)"
	for bhead in $bheads ; do
	    bhash=$(echo $bhead | cut -d " " -f 1)
	    bref=$(echo $bhead | cut -d " " -f 2)
	    [ "$bref" != "$ref" ] && continue
	    oldshort=$(git-rev-parse --short $bhash)
	    case $type in
		tags)
		    base=$newhash;;
		heads)
		    base=$(git merge-base $bref $bhash);;
	    esac
	    if [ "$base" != $bhash ] ; then
		[ -n "$header" ] && echo $header && header=
		if [ -z "$force" ] ; then
		    nonff=t
		    echo " ! [rejected] $bshort -> $bshort (non-fast forward)"
		else
		    changed=t
		    echo " + $oldshort...$newshort $bshort -> $bshort (forced update)"
		fi
		continue 2
	    fi
	    if [ "$newhash" != "$bhash" ] ; then
		changed=t
		[ -n "$header" ] && echo $header && header=
		echo "   $oldshort..$newshort $bshort -> $bshort"
	    elif [ -n "$verbose" ] ; then
		[ -n "$header" ] && echo $header && header=
		echo " = [up to date] $bshort -> $bshort"
	    fi
	    continue 2
	done
    fi
    [ -n "$header" ] && echo $header && header=
    echo " * [$newtext] $bshort -> $bshort"
    changed=t
done

[ -n "$full" ] && bases= && [ -n "$bbases" ] && changed=t

[ -n "$nonff" ] && die "error: failed to push some refs to $remoteurl"
[ -z "$changed" ] && die "Everything up-to-date"
[ -n "$bases" ] && bases="--not$LF$bases"

git bundle create $remoteurl $refs $bases
[ "$remote" != "$remoteurl" ] && git fetch -q "$remote"
exit 0

Hi,

On Tue, 9 Dec 2008, Santi Béjar wrote:

The basic idea is:

- Easily create bundles with the current branch.
- Be able to push to defined bundles in remote.<remote>.url
- Only add new objects by default (do not lose objects)

That is probably not what people need.  Usually, when bundles are sent 
around, you need _incremental_ bundles.  IOW if you already have a bundle, 
you want to create a new bundle that contains everything that is new, _in 
addition_ to the existing bundle.

while [ $# != 0 ] ; do

Heh, I did not realize just how _used_ I got to the conventions in Git's 
shell programming, until I thought "Should this not use 'test' instead 
of brackets?"

while [ $# != 0 ] ; do
    refs="$refs$LF$1" && shift
done

That is equivalent to refs="$*", no?

Anyway, I found reading your shell script quite hard, because of excessive 
use of brackets and single line && chains (which lack proper error 
handling, BTW).

Ciao,
Dscho

First, thanks for the comments.

2008/12/9 Johannes Schindelin [off-list ref]:

Hi,

On Tue, 9 Dec 2008, Santi Béjar wrote:

quoted

The basic idea is:

- Easily create bundles with the current branch.
- Be able to push to defined bundles in remote.<remote>.url
- Only add new objects by default (do not lose objects)

That is probably not what people need.  Usually, when bundles are sent
around, you need _incremental_ bundles.

I do not find convenient strictly incremental bundles, because then
you (or the other people) needs to fetch every single bundle. What I
do is add new objects until the bundle is too big and then create a
bundle with a new base. This way you don't have to worry if the other
person has applied the last bundle or not.

 IOW if you already have a bundle,
you want to create a new bundle that contains everything that is new, _in
addition_ to the existing bundle.

quoted

while [ $# != 0 ] ; do

Heh, I did not realize just how _used_ I got to the conventions in Git's
shell programming, until I thought "Should this not use 'test' instead
of brackets?"

I don't have problems either way, I'll change to follow Git's conventions.

quoted

while [ $# != 0 ] ; do
    refs="$refs$LF$1" && shift
done

That is equivalent to refs="$*", no?

Almost, IFS is set to line-feed so I needed to put $LF instead of spaces.

Anyway, I found reading your shell script quite hard, because of excessive
use of brackets and single line && chains (which lack proper error
handling, BTW).

I've tried to catch errors, but maybe not enough.

Santi

"Santi Béjar" [off-list ref] writes:

I do not find convenient strictly incremental bundles, because then
you (or the other people) needs to fetch every single bundle. What I
do is add new objects until the bundle is too big and then create a
bundle with a new base. This way you don't have to worry if the other
person has applied the last bundle or not.

You both have good points.  I sort of tend to side with your argument from
convenience point of view, if only because that resembles the way how
people traditionally arrange incremental backups "a full dump on Sunday
night, and every day incremental relative to the last full dump".  Dscho's
suggestion is akin to "a full dump on Sunday night, and every day
incremental relative to the previous day".  Both form obviously can
recreate the same contents, but often "incremental since the last full
synchronization point", even though it may make bigger dumps, is easier to
handle for humans.

quoted

 IOW if you already have a bundle,
you want to create a new bundle that contains everything that is new, _in
addition_ to the existing bundle.

quoted

quoted

while [ $# != 0 ] ; do

Heh, I did not realize just how _used_ I got to the conventions in Git's
shell programming, until I thought "Should this not use 'test' instead
of brackets?"

Now I see you are improving ;-)

quoted

quoted

while [ $# != 0 ] ; do
    refs="$refs$LF$1" && shift
done

That is equivalent to refs="$*", no?

Almost, IFS is set to line-feed so I needed to put $LF instead of spaces.

If $IFS is set to LF, "$*" will be $1, $2, $3 concatenated with LF in
between.  The first character in $IFS is used for that purpose..