Re: [BUG] Git push sends too much data unnecessarily

Rajiv Sharma [off-list ref] writes:

Thank you for filling out a Git bug report!
Please answer the following questions to help us understand your issue.

What did you do before the bug happened? (Steps to reproduce your issue)

I tried to create a new branch pointing to the commit which was the
ancestor of the current branch (i.e. HEAD~1) and pushing it to the
remote. Since the commit was already known to the server, I expected
the push to be kind of no-op since it's simply creating a new pointer.
However the push ended up taking 10+ minutes. Since I was running with
the `--verbose` flag, I realised that the push ended up sending
multiple GBs worth of data just for creating a new branch on an
existing commit already known to the remote. After some
experimentation, I managed to find an easy repro for this issue:

Clone a non-empty repo from some remote (e.g. git clone
https://SERVER_HOSTNAME/repo_name.git) in two locations, `primary` and
`secondary` and ensure that both have the same branch checked out.
Navigate to the `primary` location and create a local commit for repo
`repo_name`. Push this commit C1 to the remote server
Navigate to the `secondary` location and try to create a new branch by
running `git push origin HEAD:refs/heads/shiny_new_branch --verbose`
(or by checking out that branch and pushing it). Note that `HEAD` here
refers to the `HEAD` commit as seen by `secondary` which in reality is
`HEAD~1` compared to the remote
If the repo had some commits on the checked out branch, you will
notice the verbose output highlighting objects being sent to the
server where there was no need to do so


To understand more about exactly how much data is sent, I ran a few
more experiments and came to the conclusion that the git client sends
HEAD commit + all ancestors of HEAD commit except the commits which
are also ancestors of some other branch / ref known to Git.
Pictorially, it can be represented as:

B1  B2       <-- HEAD
*      *         (sent)
|       |
*       *         (sent)
|        |
*        *        (sent)
|      /
|    /
*                  (NOT sent)
|
*                  (NOT sent)

This explains the multi GB push in my case because I was working on a
long standing branch with lots of commits. Initially I assumed this
was a server problem but then realised that in the push path the
server just advertises refs and where they point and it's the client
that does the negotiation. I think the bug exists somewhere in the
negotiation logic but I am not sure.

Thanks for the detailed explanation. I don't think this is a bug per-se,
but that doesn't mean this isn't something we can't discuss and
potentiall optimize

To reiterate my understanding, I did a quick local PoC:

$ git init remote
$ git -C remote config set receive.denyCurrentBranch ignore
$ git -C remote commit --allow-empty -m "C1"
$ git -C remote commit --allow-empty -m "C2"
$ git -C remote commit --allow-empty -m "C3"

$ git clone remote/ base1
$ git clone remote/ base2

$ git -C base1 commit --allow-empty -m "C4"
$ git -C base1 push -f --verbose
Pushing to /tmp/remote/
Enumerating objects: 1, done.
Counting objects: 100% (1/1), done.
Writing objects: 100% (1/1), 704 bytes | 704.00 KiB/s, done.
Total 1 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
To /tmp/remote/
   78c400c..affbad8  master -> master
updating local tracking ref 'refs/remotes/origin/master'

$ git -C base2 push -f --verbose origin HEAD:refs/heads/fun
Pushing to /tmp/remote/
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 16 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 1.98 KiB | 1.98 MiB/s, done.
Total 4 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
To /tmp/remote/
 * [new branch]      HEAD -> fun
updating local tracking ref 'refs/remotes/origin/fun'

What you're stating about and can be easily seen here is that while
pushing C4 from base1 only transferred one object, pushing HEAD from
base2 (which is C4~1), pushes 4 objects.

After base1 creates C4 and pushes:
==================================
remote:     C1 --- C2 --- C3 --- C4 (master)

base1:      C1 --- C2 --- C3 --- C4 (master, origin/master)
                                  ^
                                  |
                         (transfers only C4)

base2:      C1 --- C2 --- C3 (master, origin/master)


When base2 pushes HEAD (=C3) to refs/heads/fun:
================================================
remote:     C1 --- C2 --- C3 --- C4 (master)
                            \
                             fun

base2:      C1 --- C2 --- C3 (master, origin/master)
                        ^
                        |
              (transfers C1, C2, C3, + tree object)
              (4 objects total)

This boils down to how Git negotiates between the client <> server.
In our case, remote will list the references it already contains. So in
our experiment, that'd be:

 - C4: affbad8

With this information, the client should find all the objects the remote
would need to satisfy the new references being pushed.

Since C4 is a reference the client (base2) knows nothing about, it
cannot find a common ancestor between the provided commit vs all commits
present within the repository itself. This is seems obvious to us, since
C4~1 is the common ancestor here, but base2 doesn't have sufficient
information to come to that conclusion.

So it sends all objects required to create the reference, in our case 4
objects, in your case GBs of data.

What did you expect to happen? (Expected behavior)

I would have expected the push to be extremely lightweight without
sending any objects to the server.


What happened instead? (Actual behavior)

Already detailed in the first section above.


What's different between what you expected and what actually happened?

The git client sends loads of data to the server when it shouldn't
have had to send anything at all.


Anything else you want to add:

Note that there are workarounds for this problem. If I do a `git pull`
and get the latest state of the repo before performing any push, this
problem doesn't occur. Nevertheless, I think it might be worthwhile to
fix this. I managed to repro this across OS (Linux, MacOS) and across
versions.

That said, I do think we can potentially optimize this, AFAIK the
negotiation phase has the server listing its refs and this is compared
to the list of refs locally present to determine all missing objects.

So any commits which are not represented by a ref, would be missed. One
way to reduce this would be for the server to also provide additional
information such as commits which are not represented by any refs. But
how many such commits? What about sampling? Finally we'd have to
consider if it is worth it.

Thanks,
Karthik