Hi list,
I have been working on a custom git server where the sha values
required to respond to a ls-refs command can occasionally be expensive
to compute.
It would be a great improvement of user experience if it was possible
to show progress to the user while this is happening.
I was hoping that this could be solved by using the already existing
sideband-all capability, but unfortunately it seems that even with
sideband-all the sideband is not available during ls-refs.
So my plan would be to start working on a patch to add sideband
support to ls-refs, unless there is
some other protocol feature that I have overlooked and that would
enable my use case.
Any pointers to either alternative solutions and/or hints to potential
pitfalls in the implementation
of my suggestion would be greatly appreciated.
Best regards,
Christian Schilling
On Sun, Jan 24, 2021 at 01:40:08PM +0100, Christian Schilling wrote:
Hi list,
I have been working on a custom git server where the sha values
required to respond to a ls-refs command can occasionally be expensive
to compute.
Almost certainly the expensive part of ls-refs is actually traversing
the references, not parsing the objects they point at and determining
their hash.
Incidentally, we had a discussion recently [1] that resulted in some
patches that make it so that ls-refs often only has to read through part
of the refs in your repository, not all of them.
It would be a great improvement of user experience if it was possible
to show progress to the user while this is happening.
It's possible that that might help, but honestly I'd be surprised if
there was a real use-case that needed it (especially after the patches
that I mentioned which should make it fast enough that you don't have to
care :-)).
Thanks,
Taylor
[1]: https://lore.kernel.org/git/20210119144251.27924-1-jacob@gitlab.com/
On Mon 25. Jan 2021 at 02:04, Taylor Blau [off-list ref] wrote:
On Sun, Jan 24, 2021 at 01:40:08PM +0100, Christian Schilling wrote:
quoted
Hi list,
I have been working on a custom git server where the sha values
required to respond to a ls-refs command can occasionally be expensive
to compute.
Almost certainly the expensive part of ls-refs is actually traversing
the references, not parsing the objects they point at and determining
their hash.
My use case is very different. I’m working on a custom git server that
creates refs as well as the objects they are referring to on demand.
More specifically the references are derived from the repo url to
implement partial cloning.
So you can do:
git clone http://repository-url.git:/any/subdir/you/want.git
The server will do something similar to git-filter-branch and
deliver the result.
Due to the way my own branch filtering implementation works this is
usually fast enough for the users not to notice what is going on,
but the fact remains that the shas are only known after the filtering
is complete.
Also my server acts as a proxy with the canonical backing repository
hosted on a different machine.
So to start filtering the proxy has to first ask the upstream for the
current state of the repo and download
any new changes. I have done a lot of optimizations to make
all of this fast and in most cases it is, but for the occasional case,
for example if a completely new repo is cloned for the first time,
it leaves the users looking at a seemingly “frozen” command prompt
not knowing if they made a mistake or are encountering some bug or
network problem.
At first I also thought about suggesting a protocol extension that allows
to start transferring pack data before all refs are known to reduce the
total time needed, but the slow cases are infrequent enough so that this
probably not worth it.
Incidentally, we had a discussion recently [1] that resulted in some
patches that make it so that ls-refs often only has to read through part
of the refs in your repository, not all of them.
I had my own struggles with the “to many refs” problem, but this is
unrelated :-)
quoted
It would be a great improvement of user experience if it was possible
to show progress to the user while this is happening.
It's possible that that might help, but honestly I'd be surprised if
there was a real use-case that needed it (especially after the patches
that I mentioned which should make it fast enough that you don't have to
care :-)).
Sorry for not being clearer about what I am trying to accomplish the first time,
I hope I convinced you that this is a very different, but real, use case.
Thanks,
Christian
Thanks,
Taylor
[1]: https://lore.kernel.org/git/20210119144251.27924-1-jacob@gitlab.com/