Re: [PATCH] read-cache: make the index write buffer size 128K

Jeff Hostetler [off-list ref] writes:

On 2/17/21 9:48 PM, Neeraj K. Singh via GitGitGadget wrote:

quoted

From: Neeraj Singh <redacted>
Writing an index 8K at a time invokes the OS filesystem and caching
code
very frequently, introducing noticeable overhead while writing large
indexes. When experimenting with different write buffer sizes on Windows
writing the Windows OS repo index (260MB), most of the benefit came by
bumping the index write buffer size to 64K. I picked 128K to ensure that
we're past the knee of the curve.
With this change, the time under do_write_index for an index with 3M
files goes from ~1.02s to ~0.72s.

[...]

quoted

  -#define WRITE_BUFFER_SIZE 8192
+#define WRITE_BUFFER_SIZE (128 * 1024)
  static unsigned char write_buffer[WRITE_BUFFER_SIZE];
  static unsigned long write_buffer_len;

[...]

Very nice.

I wonder if we gain more by going say 4M buffer size or even larger?

Is this something we can make the system auto-tune itself?  This is
not about reading but writing, so we already have enough information
to estimate how much we would need to write out.

Thanks.