Re: [PATCH v12 14/17] btrfs: send: write larger chunks when using stream v2

On Thu, Nov 18, 2021 at 04:50:37PM +0100, David Sterba wrote:

On Wed, Nov 17, 2021 at 12:19:24PM -0800, Omar Sandoval wrote:

quoted

From: Omar Sandoval <redacted>

The length field of the send stream TLV header is 16 bits. This means
that the maximum amount of data that can be sent for one write is 64k
minus one. However, encoded writes must be able to send the maximum
compressed extent (128k) in one command. To support this, send stream
version 2 encodes the DATA attribute differently: it has no length
field, and the length is implicitly up to the end of containing command
(which has a 32-bit length field). Although this is necessary for
encoded writes, normal writes can benefit from it, too.

Also add a check to enforce that the DATA attribute is last. It is only
strictly necessary for v2, but we might as well make v1 consistent with
it.

For v2, let's bump up the send buffer to the maximum compressed extent
size plus 16k for the other metadata (144k total).

I'm not sure we want to set the number like that, it feels quite
limiting for potential compression enhancements.

This is all we need for now, but we can always raise it in the future. I
amended the protocol and the progs send parsing code to assume no hard
limit.

quoted

Since this will most
likely be vmalloc'd (and always will be after the next commit), we round
it up to the next page since we might as well use the rest of the page
on systems with >16k pages.

Would it work also without the virtual mappings? For speedup it makes
sense to use vmalloc area, but as a fallback writing in smaller portions
or page by page eventually should be also possible. For that reason I
don't think we should set the maximum other than what fits to 32bit
number minus some overhead.

I think you're saying that we could allocate a smaller buffer and do
smaller reads that we immediately write to the send pipe/file? So
something like:

send_write() {
	write_tlv_metadata_to_pipe();
	while (written < to_write) {
		read_small_chunk();
		write_small_chunk_to_pipe();
		written += size_of_small_chunk();
	}
}

And from the protocol's point of view, it's still one big command,
although we didn't have to keep it all in memory at once.

If I'm understanding correctly, then yes, I think that's something we
could do eventually. And my description of v2 allows this:

-#define BTRFS_SEND_BUF_SIZE SZ_64K
+/*
+ * In send stream v1, no command is larger than 64k. In send stream v2, no limit
+ * should be assumed.
+ */
+#define BTRFS_SEND_BUF_SIZE_V1 SZ_64K

Although receive would have to be more intelligent about reading huge
commands.