Re: http protocol, cloning git.git, fails (too many open files)

merlyn@stonehenge.com (Randal L. Schwartz) writes:

    $ mkdir foo
    $ cd foo
    $ cg-clone http://www.kernel.org/pub/scm/git/git.git git.new
    Fetching head...
    Fetching objects...
    progress: 1 objects, 191 bytes
progress: 2 objects, 519 bytes
progress: 3 objects, 5851 bytes
progress: 4 objects, 7671 bytes
...

    progress: 66 objects, 187452 bytes

And at this point, it hung.  I had to ^C.

This sounds like the problem the patch Pasky posted last night
addresses.

However, one thing puzzles me.  I just tried to reproduce it by
doing this:

	$ rm -fr git-http
        $ ulimit -n 16
        $ git clone http://www.kernel.org/pub/scm/git/git.git git-http

and it did not fail on my Linux box.  This is from the tip of
the master tree (essentially the same as 0.99.9h aka 1.0rc1),
built without Pasky's patch.  Maybe the number of in-transit
requests in your case is much higher with your connection while
waiting for a packfile to arrive than my setup.

Could you please try with the patch Nick posted (which covers
less) and then with Pasky's patch?

On Sat, 2005-11-12 at 15:21 -0800, Junio C Hamano wrote:
[...]

However, one thing puzzles me.  I just tried to reproduce it by
doing this:

	$ rm -fr git-http
        $ ulimit -n 16
        $ git clone http://www.kernel.org/pub/scm/git/git.git git-http

and it did not fail on my Linux box...

I've never seen it on Linux, but I do see it on NetBSD every time
with ulimit -n 64 (the default):

Could you please try with the patch Nick posted (which covers
less) and then with Pasky's patch?

Neither patch fixes the 'too many open files' error, but Pasky's
patch produces this debugging output:

[...]
progress: 46 objects, 105660 bytes
error: Couldn't create temporary
file .git/objects/ee/baf3aaffa72996ac8bdbc8e9498814cd99f506.temp
for .git/objects/ee/baf3aaffa72996ac8bdbc8e9498814cd99f506: Too many
open files

progress: 48 objects, 108247 bytes
error: fd leakage in release: 9
error: fd leakage in release: 11
error: fd leakage in release: 3
error: fd leakage in release: 13
error: fd leakage in release: 6
error: fd leakage in release: 12
error: fd leakage in release: 14
error: fd leakage in release: 8
error: fd leakage in release: 16
error: fd leakage in release: 15
error: fd leakage in release: 10
error: fd leakage in release: 5
error: fd leakage in release: 21
error: fd leakage in release: 17
error: fd leakage in release: 22
error: fd leakage in release: 19
error: fd leakage in release: 20
error: fd leakage in release: 25
error: fd leakage in release: 26
error: Could not read e8700aa756a3296ede4403db349dc64586935139
Waiting for
http://www.kernel.org/pub/scm/git/git.git/objects/dd/d5823df728bf213425533dcd2e47d61cbccf9e
cg-fetch: objects fetch failed
cg-clone: fetch failed

On Sun, 13 Nov 2005, walt wrote:

On Sat, 2005-11-12 at 15:21 -0800, Junio C Hamano wrote:
[...]

quoted

However, one thing puzzles me.  I just tried to reproduce it by
doing this:

	$ rm -fr git-http
        $ ulimit -n 16
        $ git clone http://www.kernel.org/pub/scm/git/git.git git-http

and it did not fail on my Linux box...

I've never seen it on Linux...

Aha!  I do indeed see it on linux with ulimit -n 64 but *only* when I
do the clone with cg-clone.  When I use 'git clone' it works just fine.

So, what does cogito do differently that accounts for more open files?

Dear diary, on Mon, Nov 14, 2005 at 01:42:33AM CET, I got a letter
where wa1ter@myrealbox.com said that...

On Sun, 13 Nov 2005, walt wrote:

quoted

On Sat, 2005-11-12 at 15:21 -0800, Junio C Hamano wrote:
[...]

quoted

However, one thing puzzles me.  I just tried to reproduce it by
doing this:

	$ rm -fr git-http
        $ ulimit -n 16
        $ git clone http://www.kernel.org/pub/scm/git/git.git git-http

and it did not fail on my Linux box...

quoted

I've never seen it on Linux...

Aha!  I do indeed see it on linux with ulimit -n 64 but *only* when I
do the clone with cg-clone.  When I use 'git clone' it works just fine.

So, what does cogito do differently that accounts for more open files?

Cogito uses the same logic and underlying GIT fetching commands for
cloning as well as for fetching, while git-clone uses some custom logic
for cloning ("dumb http clone"). (In fact, is there a point in carrying
it further now that git-http-fetch got as smart as it got? You'll have
less code, and more importantly see bugs in the fetchers otherwise only
I see with Cogito. ;-)

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.

On Mon, 14 Nov 2005, Petr Baudis wrote:

Date: Mon, 14 Nov 2005 02:09:40 +0100
From: Petr Baudis <redacted>
Cc: git@vger.kernel.org
Newsgroups: gmane.comp.version-control.git
Subject: Re: http protocol, cloning git.git, fails (too many open files)

Dear diary, on Mon, Nov 14, 2005 at 01:42:33AM CET, I got a letter
where wa1ter@myrealbox.com said that...

quoted

So, what does cogito do differently...

Cogito uses the same logic and underlying GIT fetching commands for
cloning as well as for fetching, while git-clone uses some custom logic
for cloning ("dumb http clone")...

BTW, I just noticed one big difference between git clone and cg-clone:

git clone stores the URL in .git/remotes/origin while cg-clone puts
it in .git/branches/origin.  When I do a cg-update after a git clone
then cg-update can't find the URL.