Intriguing error with Git 1.6.3.1: Too many open files

We use Git (1.6.3.1) exclusively here at Box.

Starting this afternoon, we started getting errors when pushing to one of
the repos. It is a very active, bare shared repo, shared among
multiple developers. The
error is:

*"error: unable to open object pack directory: ./objects/pack: Too many
open files"

Things I tried/observed to resolve it:

1) ulimit -H = unlimited, when run as root (original error seems
misleading?). I also killed other proceses that had some open files, and
brought down the total count of open files (roughly lsof | wc -l) to <1000.

2) I can push to several other repos on the same machine just fine

3) git fsck, git gc, git-branch and git-status return the same error

4) I copied the repo (using scp -r, git-clone failed) to a different
machine, but got the same errors when pushing to it

5) I asked #git and the suggestion was that the repo is somehow corrupt and
I should re-push to it

One more thing worth mentioning: Gerrit (code review) uses the same repo
(lots of stuff in refs/changes)

If the repo is indeed not recoverable, I know how to restore it. But I'd
rather fix this if possible. Wondering if it is worth pursuing and what
might be wrong?

Thanks.
-Soham

Hi,

Soham Mehta wrote:

*"error: unable to open object pack directory: ./objects/pack: Too many
open files"

Do current git versions (v1.7.4.2 or later) behave the same way?

Sorry for the trouble,
Jonathan

On Tue, Jan 10, 2012 at 7:09 PM, Jonathan Nieder [off-list ref] wrote:

Hi,

Soham Mehta wrote:

quoted

*"error: unable to open object pack directory: ./objects/pack: Too many
open files"

Do current git versions (v1.7.4.2 or later) behave the same way?

Sorry for the trouble,
Jonathan

I haven't played with versions yet. Are you alluding that #2 on this
list may have addressed it?
https://raw.github.com/gitster/git/master/Documentation/RelNotes/1.7.4.2.txt

 * We used to keep one file descriptor open for each and every packfile
   that we have a mmap window on it (read: "in use"), even when for very
   tiny packfiles.  We now close the file descriptor early when the entire
   packfile fits inside one mmap window.


The corrupt repo does have a 1000+ pack files, anywhere from 650M to 2.5K.

Soham Mehta wrote:

I haven't played with versions yet.

For almost all purposes, the latest stable version should work fine
(v1.7.8.3 at the moment).  The people on this list tend to be very
keen to hear about cases where that's not true --- if people are stuck
using old versions because of regressions, that means we're failing.

Are you alluding that #2 on this
list may have addressed it?
https://raw.github.com/gitster/git/master/Documentation/RelNotes/1.7.4.2.txt

Not exactly.  I was thinking of commit c7934306 (Limit file
descriptors used by packs, 2011-02-28).

In git time, 1.6.3.1 is ancient history.  There are many little fixes
and performance improvements you are missing out on.

Just my two cents,
Jonathan