Re: [PATCH 7/9] fetch: fetch submodules in parallel

On Fri, Aug 28, 2015 at 11:35 AM, Jeff King [off-list ref] wrote:

On Fri, Aug 28, 2015 at 11:27:04AM -0700, Junio C Hamano wrote:

quoted

quoted

But for commands that show progress like "git clone", "git checkout",
and "git fetch", it does not work well at all.  They provide output
that updates itself by putting a carriage return at the end of each
chunk of output, like this:

 remote: Finding sources:  11% (18/155)           \r
 remote: Finding sources:  12% (19/155)           \r

With multiple commands producing such output, they will overwrite each
other's lines, producing a mixture that is confusing and unuseful.

That example also illustrates why it is not a useful to buffer all
of these lines and showing them once.

I think Jonathan's point is that you could pick _one_ active child to
show without buffering, while simultaneously buffering everybody else's
output. When that finishes, pick a new active child, show its buffer,
and then start showing its output in realtime. And so on.

or better yet, pick that child with the most progress (i.e. flush all finished
children and then pick the next active child), that would approximate
the progress in the output best, as it would reduce the hidden
buffered progress.

So to an observer, it would look like a serial operation, but subsequent
operations after the first would magically go much faster (because
they'd been working and buffering in the background).

And that doesn't require any additional IPC magic (though I am not sure
how we get progress in the first place if the child stderr is a
pipe...).

Moving the contents from the pipe to a strbuf buffer which we can grow
indefinitely
(way larger than pipe limits, but the output of a git fetch should be
small enough for that).

-Peff