Re: [PATCHv3 02/11] run-command: report failure for degraded output just once

Johannes Sixt [off-list ref] writes:

I think that a scenario where A and B are communicating is rather
far-fetched. We are talking about parallelizing independent tasks. I
would not worry.

I wouldn't worry too much if this were merely a hack that is only
applicable to submodules, but I do not think it is a healthy
attitude to dismiss potential problem as far-fetched without
thinking things through, when you are designing what goes into
the run-command API.

I'd grant you that a complete deadlock is unlikely to be a problem
on its own.  Somewhere somebody will eventually time out and unblock
the deadlock anyway.

But the symptom does not have to be as severe as a total deadlock to
be problematic.  If we block B (and other tasks) by not reading from
them quickly because we are blocked on reading from A, which may
take forever (in timescale of B and other tasks) to feed us enough
to satisfy strbuf_read_once(), we are wasting resource by spawning B
(and other tasks) early when we are not prepared to service them
well, on both our end and on the other side of the connection.

By the way, A and B do not have to be directly communicating to
deadlock.  The underlying system, either the remote end or the local
end or even a relaying system in between (think: network) can
throttle to cause the same symptom without A and B knowing (which
was the example I gave).

On Wed, Nov 04, 2015 at 01:01:53PM -0800, Junio C Hamano wrote:

But the symptom does not have to be as severe as a total deadlock to
be problematic.  If we block B (and other tasks) by not reading from
them quickly because we are blocked on reading from A, which may
take forever (in timescale of B and other tasks) to feed us enough
to satisfy strbuf_read_once(), we are wasting resource by spawning B
(and other tasks) early when we are not prepared to service them
well, on both our end and on the other side of the connection.

I'm not sure I understand this line of reasoning. It is entirely
possible that I have not been paying close enough attention and am
missing something subtle, so please feel free to hit me with the clue
stick.

But why would we ever block reading from A? If poll() reported to us
that "A" is ready to read, and we call strbuf_read_once(), we will make
a _single_ read call (which was, after all, the point of adding
strbuf_read_once in the first place).

So even if descriptor "A" isn't non-blocking, why would we block? Only
if the OS told us we are ready to read via poll(), but we are somehow
not (which, AFAIK, would be a bug in the OS).

So I'm not sure I see why we need to be non-blocking at all here, if we
are correctly hitting poll() and doing a single read on anybody who
claims to be ready (rather than trying to soak up all of their available
data), then we should never block, and we should never starve one
process (even without blocking, we could be in a busy loop slurping from
A and starve B, but by hitting the descriptors in round-robin for each
poll(), we make sure they all progress).

What am I missing?

-Peff