upload-pack is slow with lots of refs

I'm creating a system where a lot of remotes constantly fetch from a
central repository for deployment purposes, but I've noticed that even
with a remote.$name.fetch configuration to only get certain refs a
"git fetch" will still call git-upload pack which will provide a list
of all references.

This is being done against a repository with tens of thousands of refs
(it has a tag for each deployment), so it ends up burning a lot of CPU
time on the uploader/receiver side.

Has there been any work on extending the protocol so that the client
tells the server what refs it's interested in?

I've been looking at turning this into a push mechanism instead of a
poll mechanism, but I've also noted that even when you one tag you
still end up listing all refs on the remote side:

    $ GIT_TRACE=1 git push origin my-new-tag -n
    trace: built-in: git 'push' 'origin' 'my-new-tag' '-n'
    trace: run_command: 'ssh' 'avar@git.example.com' 'git-receive-pack
'\''/gitroot/example.git'\'''
    nohup: redirecting stderr to stdout
    To ssh://avar@git.example.com/gitroot/example.git
     * [new tag]         my-new-tag -> my-new-tag

Which seems like a lot of superfluous work when it presumably only
needs to check if there's a remote "my-new-tag" tag which conflicts
with what you're pushing..