Exec upload-pack on remote with what parameters to get direntries.

Hi,

I've got a custom ssh library which I use to make a connection to a
git server like www.github.com, user stefbon.

Now I want to get the direntries of a remote repo, and I know I have
to use upload-pack for that, but with what parameters?

I want to use the outcome to make a fuse fs, user can browse the
files. Possibly the user can also view the contents.

Stef

On Sat, Aug 28, 2021 at 02:56:17PM +0200, Stef Bon wrote:

I've got a custom ssh library which I use to make a connection to a
git server like www.github.com, user stefbon.

Now I want to get the direntries of a remote repo, and I know I have
to use upload-pack for that, but with what parameters?

I want to use the outcome to make a fuse fs, user can browse the
files. Possibly the user can also view the contents.

The protocol used by upload-pack is described in
Documentation/technical/pack-protocol.txt, but in short: I don't think
it will do what you want.

There is no operation to list the tree contents, for example, nor really
even a good way to fetch a single object. The protocol is geared around
efficiently transferring slices of history, so it is looking at sets of
reachable objects (what the client is asking for, and what it claims to
have).

You might be able to cobble something together with shallow and partial
fetches. E.g., something like:

  git clone --depth 1 --filter=blob:none --single-branch -b $branch

is basically asking to send only a single commit, plus all of its trees,
but no blobs. From there you could parse the tree objects to assemble a
directory listing. Possibly with a tree:depth filter you could even do
it iteratively.

Some hosts offer a separate API that would give you a much nicer
interface. E.g., GitHub has:

  https://docs.github.com/en/rest/reference/git#trees

But of course that won't work with GitLab, etc, and you'd have to
implement against the API for each hosting provider.

-Peff

Hi,

thank you for the answer.

I understand that the core of git is to make people work together when
writing code.
To get a tree of the source files is not directly part of that, but
pure informational. That is also the intent of my fuse fs: provide the
user information about the source files.

Now I have a working ssh connection to the server, and open a channel
for running the upload-pack on the server using the exec channel
request:

https://datatracker.ietf.org/doc/html/rfc4254#section-6.5

So in my program I do not have to do something like:

ssh -x git@server "git-upload-pack 'simplegit-progit.git'"

It is only the sending of an exec message with the right command.
Via the SSH_MSG_CHANNEL_DATA message the server will return the
output. In my program I have to write a parser to get the
tree/direntries.

Now you suggest the git clone --depth 1 --filter=blob:none
--single-branch -b $branch
command. How does that look when writing it in lowlevel git messages
as described in

https://git-scm.com/book/en/v2/Git-Internals-Transfer-Protocols

?
I'm programming at this low level, so I have to write the messages to
send to the server myself.

And you mention the api github has for a git tree object. But git2 has
already the git_tree object?

Stef

My project  by the way is: https://github.com/stefbon/OSNS

On Tue, Aug 31, 2021 at 08:38:39AM +0200, Stef Bon wrote:

So in my program I do not have to do something like:

ssh -x git@server "git-upload-pack 'simplegit-progit.git'"

It is only the sending of an exec message with the right command.
Via the SSH_MSG_CHANNEL_DATA message the server will return the
output. In my program I have to write a parser to get the
tree/direntries.

Now you suggest the git clone --depth 1 --filter=blob:none
--single-branch -b $branch
command. How does that look when writing it in lowlevel git messages
as described in

https://git-scm.com/book/en/v2/Git-Internals-Transfer-Protocols

?
I'm programming at this low level, so I have to write the messages to
send to the server myself.

You'll have to read the documentation I pointed to earlier:

  https://github.com/git/git/blob/master/Documentation/technical/pack-protocol.txt

In short: the server tells you which refs it has and what they point to,
then the client says which objects it wants and which objects it has,
and then the server sends a packfile. The flow of the protocol and the
format of the messages is laid out there.

You might also set GIT_TRACE_PACKET=1 in your environment and try
running some Git commands. They will show you what's being said on the
wire, up until the packfile is sent (decoding the packfile itself is a
whole other story).

And you mention the api github has for a git tree object. But git2 has
already the git_tree object?

If you mean libgit2, then yes, it has a git_tree struct. Just like we
have internally within regular Git. But those are for accessing _local_
objects, that have already been fetched.

You could build a fuse filesystem around a local Git repository pretty
easily, either by using libgit2 or around tools like "git ls-tree" and
"git cat-file". But if your purpose is to access a remote one without
downloading all of the objects first, then no, Git does not expose any
of the endpoints you'd need remotely (but provider-specific APIs like
GitHub's do).

-Peff

Op di 31 aug. 2021 om 09:07 schreef Jeff King [off-list ref]:

On Tue, Aug 31, 2021 at 08:38:39AM +0200, Stef Bon wrote:

You might also set GIT_TRACE_PACKET=1 in your environment and try
running some Git commands. They will show you what's being said on the
wire, up until the packfile is sent (decoding the packfile itself is a
whole other story).

Yes that will give me the insight I need.
I will come back when it comes to decoding the packfile.

Thanks,
Stef

On Tue, Aug 31 2021, Stef Bon wrote:

Op di 31 aug. 2021 om 09:07 schreef Jeff King [off-list ref]:

quoted

On Tue, Aug 31, 2021 at 08:38:39AM +0200, Stef Bon wrote:

quoted

You might also set GIT_TRACE_PACKET=1 in your environment and try
running some Git commands. They will show you what's being said on the
wire, up until the packfile is sent (decoding the packfile itself is a
whole other story).

Yes that will give me the insight I need.
I will come back when it comes to decoding the packfile.

Aside from the "here's how you can do it", you haven't said why you'd
like to do such "online" browsing of the repository.

I'd think that even for something that e.g. implements a file browser
with magic git-remote support (think GNOME VFS-like), what you'd want to
do in the background would be to do a "clone", although a clone with
some combination of --single-branch, --no-tags, and perhaps --depth and
the filters discussed upthread.

It will take the same time to get the pack, but once you do you can use
libgit2, git's plumbing etc. to do really fast browsing/wildcarding
etc. of the entries locally.

So is there a real performance or other use-case for wanting to do this,
or does it just come down a lack of nice a "one-shot" API for "list
remote files?".

In any case, on the topic of clever things you can (ab)use to do this,
some remotes support running "git archive" for you. Notably GitHub
doesn't, but GitLab does. Please don't take this as an endorsement to
run this command "in production"

    $ time (git archive --format=tar  --remote=git@gitlab.com:git-vcs/git.git --prefix=t/t4018/ HEAD:t/t4018 | tar -tf- | head -n 3)
    t/t4018/
    t/t4018/README
    t/t4018/bash-arithmetic-function

    real    0m1.545s

I idly wonder if there's a want/need for a file listing API whether
doing so via the tar/zip format wouldn't be a more viable & widely
supported thing than expecting everyone to come up with their own git
packfile decoders. I.e. if we just supported some option to create
all-empty dummy files via "git archive" this could be even better as a
dummy file listing API. Right now this (ab)use of it requires
e.g. sending ~10MB of t/'s content just to list everything in the t/
directory.