Re: [MAKESHIFT PATCH] Cope better with a _lot_ of packs

"Shawn O. Pearce" [off-list ref] writes:

Johannes Schindelin [off-list ref] wrote:

quoted

You might end up with a situation where you have tons of pack files, e.g.
when using hg2git.  In this situation, all kinds of operations may 
end up with a "too many files open" error.  Let's recover gracefully from 
that.

Signed-off-by: Johannes Schindelin <redacted>

Yea, this looks right to me.  JGit had a similar problem and now
maintains a core.packedGitOpenFiles parameter, set to 128 by default,
as one of the rules it uses to limit the size of its pack cache area.

quoted

diff --git a/sha1_file.c b/sha1_file.c
index 28bd908..bd5edd8 100644
--- a/sha1_file.c
+++ b/sha1_file.c

@@ -720,6 +720,8 @@ static int open_packed_git_1(struct packed_git *p)
 		return error("packfile %s index unavailable", p->pack_name);
 
 	p->pack_fd = open(p->pack_name, O_RDONLY);
+	while (p->pack_fd < 0 && errno == EMFILE && unuse_one_window(p, -1))
+		p->pack_fd = open(p->pack_name, O_RDONLY);
 	if (p->pack_fd < 0 || fstat(p->pack_fd, &st))
 		return -1;

I wonder if this use of unuse_one_window() can be made more clever to drop
windows from the same packfile first.  If you have 200 packfiles with 2
windows open for each one when you ran out the file descriptors, you do
not want to drop one window from each packfile (which would not give you
any free file descriptor) and then finally release another window (at
which point we have one packfile with no window and we can close the
descriptor).

But at the same time I do not know if it matters in practice.  Presumably
you have too many packfiles because many of them are very small.