Hi Martin,

On Mon, Jan 05, 2026 at 11:45:41PM +0000, Martin Fick wrote:

OK, after discovering the strace -r and -T options, I have determined that
the 29K writes were all very fast in themselves. However, most of the
writes seem to follow each other with no other system calls in between.
This explains why it looks like the writes are slow, even though they aren't.

If I tally up the time between the previous system call, and each write(),
it adds up to the bulk of the time (4mins out of 4m15s) that it takes to
pack refs. This tells me that no visible I/O or system calls are the problem,
but rather that the program itself is taking a long time between writes.
I very much doubt that this is heavy CPU time, but rather I am going to 
guess that this is hidden system time spent accessing mmaped memory. 
Could it be really slow reading the packed-refs file? I can see the 
packed-refs file is mmaped() before the writes start, and then 
munmapped after the writes are completed. If I had to guess, that likely
means that the packed-refs file is being read in small increments by the 
kernel via mmap, and that is what is making things very slow over NFS. 

I wouldn't be surprised if NFS was the culprit. At GitLab we found it to
be a constant source of issues, which is why we eventually sunsetted the
use of it completely. Do you use any special flags for mounting the NFS
filesystem?

My alternative theory, is that each ref is being looked up via a binary 
search, but I don't think git does this?

Did you try using perf(1) to profile the process and generate a flame
graph from it? That should likely make it immediately obvious where Git
is spending all of its time.

Patrick