Re: [PATCH v6 06/13] pkt-line: add functions to read/write flush terminated packet streams

Stefan Beller [off-list ref] writes:

So the API provided by these read/write functions is intended
to move a huge chunks of data. And as it puts the data on the wire one
packet after the other without the possibility to intervene and e.g. send
a side channel progress bar update, I would question the design of this.

Hmph, I didn't think about it.

But shouldn't one be able to set up sideband and channel one such
large transfer on one band, while multiplexing other payload on
other bands?

If I understand correctly this will be specifically  used for large
files locally,
so e.g. a file of 5 GB (such as a virtual machine tracked in Git), would
require about 80k packets.

What is wrong about that?  4*80k = 320kB overhead for length fields
to transfer 5GB worth of data?  I do not think it is worth worrying
about it.

But I am more surprised by seeing that "why not a single huge
packet" suggestion immediately after you talked about "without the
possibility to intervene".  They do not seem to be remotely related;
in fact, they are going into opposite directions.

Puzzled.

On Thu, Aug 25, 2016 at 3:31 PM, Junio C Hamano [off-list ref] wrote:

What is wrong about that?  4*80k = 320kB overhead for length fields
to transfer 5GB worth of data?  I do not think it is worth worrying
about it.

But I am more surprised by seeing that "why not a single huge
packet" suggestion immediately after you talked about "without the
possibility to intervene".  They do not seem to be remotely related;
in fact, they are going into opposite directions.

Puzzled.

Stefan's argument to me is thus "If we're already going to ignore
sideband packets here, why not go all the way and make variable length
packets and send a single packet of a maximum length? Doing thus will
solve some set of future problems nicely and makes this code easier."

I'm not sure I agree myself, but that's the logic as I understand it.

On Thu, Aug 25, 2016 at 5:55 PM, Jacob Keller [off-list ref] wrote:

On Thu, Aug 25, 2016 at 3:31 PM, Junio C Hamano [off-list ref] wrote:

quoted

What is wrong about that?  4*80k = 320kB overhead for length fields
to transfer 5GB worth of data?  I do not think it is worth worrying
about it.

But I am more surprised by seeing that "why not a single huge
packet" suggestion immediately after you talked about "without the
possibility to intervene".  They do not seem to be remotely related;
in fact, they are going into opposite directions.

Puzzled.

Stefan's argument to me is thus "If we're already going to ignore
sideband packets here, why not go all the way and make variable length
packets and send a single packet of a maximum length? Doing thus will
solve some set of future problems nicely and makes this code easier."

I'm not sure I agree myself, but that's the logic as I understand it.

Yeah. To me it seems this design explicitly makes it hard for side bands.
As we do not need sidebands for local transfers, this is fine for sure.

(If we wanted to make it sideband friendly, I'd expect you could register
callbacks for either all packets or for a given sideband until the next
flush comes.)

So as hinted by this design, we want a protocol that
* doesn't care about sidebands
* cares about large data (hence maybe throughput)
* has easy/clean interface

And one large packet would suffice for these three points as well
and additionally has benefits for the network stuff.

The 320kB additional transmission are negligible overhead, so I was not
concerned about the size, but rather the code being bloated, i.e. we need
one layer of additional code to cope with the repetitive packets.

---
My background is mostly submodule related, and whenever I come up
with a shiny novel idea that would help submodules tremendously, someone
(mostly Peff) comes along and suggests a broader more generic thing, that
works just as well for submodules but is applicable to all of Git.

So I picked up that way of thinking: If we write code here, that helps with a
very special niche of things, can we make so, that the rest also benefits?

I may not understand all the requirements in this case, but to me it looks like
the "one packet" approach covers all requirements, but has the huge potential
to make other parts better in the long run.
-- 

Thanks,
Stefan

On Fri, Aug 26, 2016 at 10:02:52AM -0700, Stefan Beller wrote:

Yeah. To me it seems this design explicitly makes it hard for side bands.
As we do not need sidebands for local transfers, this is fine for sure.

(If we wanted to make it sideband friendly, I'd expect you could register
callbacks for either all packets or for a given sideband until the next
flush comes.)

So as hinted by this design, we want a protocol that
* doesn't care about sidebands
* cares about large data (hence maybe throughput)
* has easy/clean interface

And one large packet would suffice for these three points as well
and additionally has benefits for the network stuff.

The 320kB additional transmission are negligible overhead, so I was not
concerned about the size, but rather the code being bloated, i.e. we need
one layer of additional code to cope with the repetitive packets.

Maybe I don't understand what you mean by "one large packet". But if you
mean sending "I am about to send you 100MB" followed by the 100MB, the
point is that the sender does not necessarily have that exact value
ahead of time. So it would want to write it in chunks.

E.g., consider a clean which replaces s/foo/bar/ in its content. It
would write out all the content up to the next "foo", then write "bar",
and repeat.

We could let each chunk be arbitrarily big. I.e., "I am about to send
you 50MB", then "here are 3 bytes", then "here are the other 50MB".

But using a fixed-size length header makes the packets easy to generate
and parse. And 64KB is small enough that senders and receivers can
easily buffer single packets, making interfaces simpler. And it's big
enough that the overhead (4 bytes per 64KB) is negligible.

Anyway. It certainly does not seem worth moving the network protocols to
a new data format. The compatibility changes would not be worth it. But
if we _were_ to do so, there are tons of efficient well-thought-out
data-marshaling formats we could use rather than inventing our own. But
I just don't see the benefit.

---
My background is mostly submodule related, and whenever I come up
with a shiny novel idea that would help submodules tremendously, someone
(mostly Peff) comes along and suggests a broader more generic thing, that
works just as well for submodules but is applicable to all of Git.

Heh. It is not always a good thing, if it derails the original purpose
of the code (and I include some of my suggestions in that). :)

The trick is to make it general enough, without getting lost in the
weeds of what _might_ happen in the future and wasting time (and
adding complexity).

-Peff