Re: [PATCH v6 06/13] pkt-line: add functions to read/write flush terminated packet streams

On Fri, Aug 26, 2016 at 10:02:52AM -0700, Stefan Beller wrote:

Yeah. To me it seems this design explicitly makes it hard for side bands.
As we do not need sidebands for local transfers, this is fine for sure.

(If we wanted to make it sideband friendly, I'd expect you could register
callbacks for either all packets or for a given sideband until the next
flush comes.)

So as hinted by this design, we want a protocol that
* doesn't care about sidebands
* cares about large data (hence maybe throughput)
* has easy/clean interface

And one large packet would suffice for these three points as well
and additionally has benefits for the network stuff.

The 320kB additional transmission are negligible overhead, so I was not
concerned about the size, but rather the code being bloated, i.e. we need
one layer of additional code to cope with the repetitive packets.

Maybe I don't understand what you mean by "one large packet". But if you
mean sending "I am about to send you 100MB" followed by the 100MB, the
point is that the sender does not necessarily have that exact value
ahead of time. So it would want to write it in chunks.

E.g., consider a clean which replaces s/foo/bar/ in its content. It
would write out all the content up to the next "foo", then write "bar",
and repeat.

We could let each chunk be arbitrarily big. I.e., "I am about to send
you 50MB", then "here are 3 bytes", then "here are the other 50MB".

But using a fixed-size length header makes the packets easy to generate
and parse. And 64KB is small enough that senders and receivers can
easily buffer single packets, making interfaces simpler. And it's big
enough that the overhead (4 bytes per 64KB) is negligible.

Anyway. It certainly does not seem worth moving the network protocols to
a new data format. The compatibility changes would not be worth it. But
if we _were_ to do so, there are tons of efficient well-thought-out
data-marshaling formats we could use rather than inventing our own. But
I just don't see the benefit.

---
My background is mostly submodule related, and whenever I come up
with a shiny novel idea that would help submodules tremendously, someone
(mostly Peff) comes along and suggests a broader more generic thing, that
works just as well for submodules but is applicable to all of Git.

Heh. It is not always a good thing, if it derails the original purpose
of the code (and I include some of my suggestions in that). :)

The trick is to make it general enough, without getting lost in the
weeds of what _might_ happen in the future and wasting time (and
adding complexity).

-Peff