Re: [RFC 1/4 v2] Implement a basic remote helper for svn in C.

Hi,

back to the pipe-topic.

On Wednesday 01 August 2012 12:42:48 Jonathan Nieder wrote:

Hi again,

Florian Achleitner wrote:

quoted

When the first line arrives at the remote-helper, it starts importing one
line at a time, leaving the remaining lines in the pipe.
For importing it requires the data from fast-import, which would be mixed
with import lines or queued at the end of them.

Oh, good catch.

The way it's supposed to work is that in a bidi-import, the remote
helper reads in the entire list of refs to be imported and only once
the newline indicating that that list is over arrives starts writing
its fast-import stream.  We could make this more obvious by not
spawning fast-import until immediately before writing that newline.

This needs to be clearly documented in the git-remote-helpers(1) page
if the bidi-import command is introduced.

If a remote helper writes commands for fast-import before that newline
comes, that is a bug in the remote helper, plain and simple.  It might
be fun to diagnose this problem:

This would require all existing remote helpers that use 'import' to be ported 
to the new concept, right? Probably there is no other..

	static void pipe_drained_or_die(int fd, const char *msg)
	{
		char buf[1];
		int flags = fcntl(fd, F_GETFL);
		if (flags < 0)
			die_errno("cannot get pipe flags");
		if (fcntl(fd, F_SETFL, flags | O_NONBLOCK))
			die_errno("cannot set up non-blocking pipe read");
		if (read(fd, buf, 1) > 0)
			die("%s", msg);
		if (fcntl(fd, F_SETFL, flags))
			die_errno("cannot restore pipe flags");
	}
	...

	for (i = 0; i < nr_heads; i++) {
		write "import %s\n", to_fetch[i]->name;
	}

	if (getenv("GIT_REMOTE_HELPERS_SLOW_SANITY_CHECK"))
		sleep(1);

	pipe_drained_or_die("unexpected output from remote helper before
fast-import launch");

	if (get_importer(transport, &fastimport))
		die("couldn't run fast-import");
	write_constant(data->helper->in, "\n");

I still don't believe that sharing the input pipe of the remote helper is 
worth the hazzle.
It still requires an additional pipe to be setup, the one from fast-import to 
the remote-helper, sharing one FD at the remote helper.
It still requires more than just stdin, stdout, stderr.

I would suggest to use a fifo. It can be openend independently, after forking 
and on windows they have named pipes with similar semantics, so I think this 
could be easily ported. 
I would suggest the following changes:
- add a capability to the remote helper 'bidi-import', or 'bidi-pipe'. This 
signals that the remote helper requires data from fast-import.

- add a command 'bidi-import', or 'bidi-pipe' that is tells the remote helper 
which filename the fifo is at, so that it can open it and read it when it 
handles 'import' commands.

- transport-helper.c creates the fifo on demand, i.e. on seeing the capability, 
in the gitdir or in /tmp.

- fast-import gets the name of the fifo as a command-line argument. The 
alternative would be to add a command, but that's not allowed, because it 
changes the stream semantics.
Another alternative would be to use the existing --cat-pipe-fd argument. But 
that requires to open the fifo before execing fast-import and makes us 
dependent on the posix model of forking and inheriting file descriptors, while 
opening a fifo in fast-import would not.