From: Omar Sandoval <redacted>
This series has three parts: new Btrfs ioctls for reading/writing
compressed data, support for sending compressed data via Btrfs send, and
btrfs-progs support for sending/receiving compressed data and writing it
with the new ioctl.
Patches 1 and 2 are VFS changes exporting a couple of helpers for checks
needed by reads and writes. Patches 3-7 are preparatory Btrfs changes
for compressed reads and writes. Patch 8 adds the compressed read ioctl
and patch 9 adds the compressed write ioctl.
The main use-case for this interface is Btrfs send/receive. Currently,
when sending data from one compressed filesystem to another, the sending
side decompresses the data and the receiving side recompresses it before
writing it out. This is wasteful and can be avoided if we can just send
and write compressed extents.
Patches 10-14 add the Btrfs send support. See the previous posting for
more details and benchmarks [1]. Patches 10-12 prepare some protocol
changes for send stream v2. Patch 13 implements compressed send. Patch
14 enables send stream v2 and compressed send in the send ioctl when
requested.
These patches are based on Dave Sterba's Btrfs misc-next branch [2],
which is in turn currently based on v5.14-rc7. Test cases are here [3].
Changes since v10 [4]:
Addressed Dave's and Nikolay's comments, mostly stylistic.
- Renamed __generic_write_checks() to generic_write_checks_count().
- Add missing count == 0 check to generic_write_checks_count().
- Used in_range() macro in btrfs_csum_one_bio().
- Renamed page_offsets variable to use_page_offsets in
btrfs_csum_one_bio().
- Removed conditional on offset increment in btrfs_csum_one_bio().
- Fixed stale reference to RWF_ENCODED in comment.
1: https://lore.kernel.org/linux-btrfs/cover.1615922753.git.osandov@fb.com/
2: https://github.com/kdave/btrfs-devel/tree/misc-next
3: https://github.com/osandov/xfstests/tree/btrfs-encoded-io
4: https://lore.kernel.org/linux-btrfs/cover.1629234193.git.osandov@fb.com/
Omar Sandoval (14):
fs: export rw_verify_area()
fs: export variant of generic_write_checks without iov_iter
btrfs: don't advance offset for compressed bios in
btrfs_csum_one_bio()
btrfs: add ram_bytes and offset to btrfs_ordered_extent
btrfs: support different disk extent size for delalloc
btrfs: optionally extend i_size in cow_file_range_inline()
btrfs: add definitions + documentation for encoded I/O ioctls
btrfs: add BTRFS_IOC_ENCODED_READ
btrfs: add BTRFS_IOC_ENCODED_WRITE
btrfs: add send stream v2 definitions
btrfs: send: write larger chunks when using stream v2
btrfs: send: allocate send buffer with alloc_page() and vmap() for v2
btrfs: send: send compressed extents with encoded writes
btrfs: send: enable support for stream v2 and compressed writes
fs/btrfs/compression.c | 12 +-
fs/btrfs/compression.h | 6 +-
fs/btrfs/ctree.h | 17 +-
fs/btrfs/delalloc-space.c | 18 +-
fs/btrfs/file-item.c | 32 +-
fs/btrfs/file.c | 68 ++-
fs/btrfs/inode.c | 911 +++++++++++++++++++++++++++++++++----
fs/btrfs/ioctl.c | 213 +++++++++
fs/btrfs/ordered-data.c | 124 ++---
fs/btrfs/ordered-data.h | 25 +-
fs/btrfs/relocation.c | 2 +-
fs/btrfs/send.c | 307 +++++++++++--
fs/btrfs/send.h | 32 +-
fs/internal.h | 5 -
fs/read_write.c | 34 +-
include/linux/fs.h | 2 +
include/uapi/linux/btrfs.h | 149 +++++-
17 files changed, 1686 insertions(+), 271 deletions(-)
The btrfs-progs patches were written by Boris Burkov with some updates
from me. Patches 1-4 are preparation. Patch 5 implements encoded writes.
Patch 6 implements the fallback to decompressing. Patches 7 and 8
implement the other commands. Patch 9 adds the new `btrfs send` options.
Patch 10 adds a test case.
Changes from v10:
- Fixed feature fallback to check for ENOTTY instead of EOPNOTSUPP.
Boris Burkov (10):
btrfs-progs: receive: support v2 send stream larger tlv_len
btrfs-progs: receive: dynamically allocate sctx->read_buf
btrfs-progs: receive: support v2 send stream DATA tlv format
btrfs-progs: receive: add send stream v2 cmds and attrs to send.h
btrfs-progs: receive: process encoded_write commands
btrfs-progs: receive: encoded_write fallback to explicit decode and
write
btrfs-progs: receive: process fallocate commands
btrfs-progs: receive: process setflags ioctl commands
btrfs-progs: send: stream v2 ioctl flags
btrfs-progs: receive: add tests for basic encoded_write send/receive
Documentation/btrfs-receive.asciidoc | 4 +
Documentation/btrfs-send.asciidoc | 16 +-
cmds/receive-dump.c | 31 +-
cmds/receive.c | 347 +++++++++++++++++-
cmds/send.c | 54 ++-
common/send-stream.c | 157 ++++++--
common/send-stream.h | 7 +
ioctl.h | 149 +++++++-
libbtrfsutil/btrfs.h | 17 +-
send.h | 19 +-
.../049-receive-write-encoded/test.sh | 114 ++++++
11 files changed, 871 insertions(+), 44 deletions(-)
create mode 100755 tests/misc-tests/049-receive-write-encoded/test.sh
--
2.33.0
From: Omar Sandoval <redacted>
Encoded I/O in Btrfs needs to check a write with a given logical size
without an iov_iter that matches that size (because the iov_iter we have
is for the compressed data). So, factor out the parts of
generic_write_check() that don't need an iov_iter into a new
generic_write_checks_count() function and export that.
Signed-off-by: Omar Sandoval <redacted>
---
fs/read_write.c | 33 ++++++++++++++++++++-------------
include/linux/fs.h | 1 +
2 files changed, 21 insertions(+), 13 deletions(-)
@@ -1633,24 +1633,16 @@ int generic_write_check_limits(struct file *file, loff_t pos, loff_t *count)return0;}-/*-*Performsnecessarychecksbeforedoingawrite-*-*Canadjustwritingpositionoramountofbytestowrite.-*Returnsappropriateerrorcodethatcallershouldreturnor-*zeroincasethatwriteshouldbeallowed.-*/-ssize_tgeneric_write_checks(structkiocb*iocb,structiov_iter*from)+/* Like generic_write_checks(), but takes size of write instead of iter. */+intgeneric_write_checks_count(structkiocb*iocb,loff_t*count){structfile*file=iocb->ki_filp;structinode*inode=file->f_mapping->host;-loff_tcount;-intret;if(IS_SWAPFILE(inode))return-ETXTBSY;-if(!iov_iter_count(from))+if(!*count)return0;/* FIXME: this is for backwards compatibility with 2.4 */
From: Omar Sandoval <redacted>
btrfs_csum_one_bio() loops over each filesystem block in the bio while
keeping a cursor of its current logical position in the file in order to
look up the ordered extent to add the checksums to. However, this
doesn't make much sense for compressed extents, as a sector on disk does
not correspond to a sector of decompressed file data. It happens to work
because 1) the compressed bio always covers one ordered extent and 2)
the size of the bio is always less than the size of the ordered extent.
However, the second point will not always be true for encoded writes.
Let's add a boolean parameter to btrfs_csum_one_bio() to indicate that
it can assume that the bio only covers one ordered extent. Since we're
already changing the signature, let's get rid of the contig parameter
and make it implied by the offset parameter, similar to the change we
recently made to btrfs_lookup_bio_sums(). Additionally, let's rename
nr_sectors to blockcount to make it clear that it's the number of
filesystem blocks, not the number of 512-byte sectors.
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Omar Sandoval <redacted>
---
fs/btrfs/compression.c | 5 +++--
fs/btrfs/ctree.h | 2 +-
fs/btrfs/file-item.c | 32 ++++++++++++++------------------
fs/btrfs/inode.c | 8 ++++----
4 files changed, 22 insertions(+), 25 deletions(-)
From: Omar Sandoval <redacted>
Currently, we only create ordered extents when ram_bytes == num_bytes
and offset == 0. However, RWF_ENCODED writes may create extents which
only refer to a subset of the full unencoded extent, so we need to plumb
these fields through the ordered extent infrastructure and pass them
down to insert_reserved_file_extent().
Since we're changing the btrfs_add_ordered_extent* signature, let's get
rid of the trivial wrappers and add a kernel-doc.
Reviewed-by: Nikolay Borisov <redacted>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Omar Sandoval <redacted>
---
fs/btrfs/inode.c | 56 +++++++++++---------
fs/btrfs/ordered-data.c | 112 +++++++++++-----------------------------
fs/btrfs/ordered-data.h | 22 ++++----
3 files changed, 76 insertions(+), 114 deletions(-)
@@ -1180,9 +1180,9 @@ static noinline int cow_file_range(struct btrfs_inode *inode,}free_extent_map(em);-ret=btrfs_add_ordered_extent(inode,start,ins.objectid,-ram_size,cur_alloc_size,-BTRFS_ORDERED_REGULAR);+ret=btrfs_add_ordered_extent(inode,start,ram_size,ram_size,+ins.objectid,cur_alloc_size,0,+0,BTRFS_COMPRESS_NONE);if(ret)gotoout_drop_extent_cache;
@@ -1843,10 +1843,11 @@ static noinline int run_delalloc_nocow(struct btrfs_inode *inode,gotoerror;}free_extent_map(em);-ret=btrfs_add_ordered_extent(inode,cur_offset,-disk_bytenr,num_bytes,-num_bytes,-BTRFS_ORDERED_PREALLOC);+ret=btrfs_add_ordered_extent(inode,+cur_offset,num_bytes,num_bytes,+disk_bytenr,num_bytes,0,+1<<BTRFS_ORDERED_PREALLOC,+BTRFS_COMPRESS_NONE);if(ret){btrfs_drop_extent_cache(inode,cur_offset,cur_offset+num_bytes-1,
@@ -1855,9 +1856,11 @@ static noinline int run_delalloc_nocow(struct btrfs_inode *inode,}}else{ret=btrfs_add_ordered_extent(inode,cur_offset,+num_bytes,num_bytes,disk_bytenr,num_bytes,-num_bytes,-BTRFS_ORDERED_NOCOW);+0,+1<<BTRFS_ORDERED_NOCOW,+BTRFS_COMPRESS_NONE);if(ret)gotoerror;}
@@ -2842,6 +2845,7 @@ static int insert_reserved_file_extent(struct btrfs_trans_handle *trans,structbtrfs_keyins;u64disk_num_bytes=btrfs_stack_file_extent_disk_num_bytes(stack_fi);u64disk_bytenr=btrfs_stack_file_extent_disk_bytenr(stack_fi);+u64offset=btrfs_stack_file_extent_offset(stack_fi);u64num_bytes=btrfs_stack_file_extent_num_bytes(stack_fi);u64ram_bytes=btrfs_stack_file_extent_ram_bytes(stack_fi);structbtrfs_drop_extents_argsdrop_args={0};
@@ -2916,7 +2920,8 @@ static int insert_reserved_file_extent(struct btrfs_trans_handle *trans,gotoout;ret=btrfs_alloc_reserved_file_extent(trans,root,btrfs_ino(inode),-file_pos,qgroup_reserved,&ins);+file_pos-offset,+qgroup_reserved,&ins);out:btrfs_free_path(path);
@@ -2942,20 +2947,20 @@ static int insert_ordered_extent_file_extent(struct btrfs_trans_handle *trans,structbtrfs_ordered_extent*oe){structbtrfs_file_extent_itemstack_fi;-u64logical_len;boolupdate_inode_bytes;+u64num_bytes=oe->num_bytes;+u64ram_bytes=oe->ram_bytes;memset(&stack_fi,0,sizeof(stack_fi));btrfs_set_stack_file_extent_type(&stack_fi,BTRFS_FILE_EXTENT_REG);btrfs_set_stack_file_extent_disk_bytenr(&stack_fi,oe->disk_bytenr);btrfs_set_stack_file_extent_disk_num_bytes(&stack_fi,oe->disk_num_bytes);+btrfs_set_stack_file_extent_offset(&stack_fi,oe->offset);if(test_bit(BTRFS_ORDERED_TRUNCATED,&oe->flags))-logical_len=oe->truncated_len;-else-logical_len=oe->num_bytes;-btrfs_set_stack_file_extent_num_bytes(&stack_fi,logical_len);-btrfs_set_stack_file_extent_ram_bytes(&stack_fi,logical_len);+num_bytes=ram_bytes=oe->truncated_len;+btrfs_set_stack_file_extent_num_bytes(&stack_fi,num_bytes);+btrfs_set_stack_file_extent_ram_bytes(&stack_fi,ram_bytes);btrfs_set_stack_file_extent_compression(&stack_fi,oe->compress_type);/* Encryption and other encoding is reserved and all 0 */
@@ -161,7 +172,8 @@ static int __btrfs_add_ordered_extent(struct btrfs_inode *inode, u64 file_offsetstructbtrfs_ordered_extent*entry;intret;-if(type==BTRFS_ORDERED_NOCOW||type==BTRFS_ORDERED_PREALLOC){+if(flags&+((1<<BTRFS_ORDERED_NOCOW)|(1<<BTRFS_ORDERED_PREALLOC))){/* For nocow write, we can release the qgroup rsv right now */ret=btrfs_qgroup_free_data(inode,NULL,file_offset,num_bytes);if(ret<0)
@@ -191,18 +205,12 @@ static int __btrfs_add_ordered_extent(struct btrfs_inode *inode, u64 file_offsetentry->qgroup_rsv=ret;entry->physical=(u64)-1;-ASSERT(type==BTRFS_ORDERED_REGULAR||-type==BTRFS_ORDERED_NOCOW||-type==BTRFS_ORDERED_PREALLOC||-type==BTRFS_ORDERED_COMPRESSED);-set_bit(type,&entry->flags);+ASSERT((flags&~BTRFS_ORDERED_TYPE_FLAGS)==0);+entry->flags=flags;percpu_counter_add_batch(&fs_info->ordered_bytes,num_bytes,fs_info->delalloc_batch);-if(dio)-set_bit(BTRFS_ORDERED_DIRECT,&entry->flags);-/* one ref for the tree */refcount_set(&entry->refs,1);init_waitqueue_head(&entry->wait);
@@ -76,6 +76,13 @@ enum {BTRFS_ORDERED_PENDING,};+/* BTRFS_ORDERED_* flags that specify the type of the extent. */+#define BTRFS_ORDERED_TYPE_FLAGS ((1UL << BTRFS_ORDERED_REGULAR) | \+(1UL<<BTRFS_ORDERED_NOCOW)|\+(1UL<<BTRFS_ORDERED_PREALLOC)|\+(1UL<<BTRFS_ORDERED_COMPRESSED)|\+(1UL<<BTRFS_ORDERED_DIRECT))+structbtrfs_ordered_extent{/* logical offset in the file */u64file_offset;
@@ -84,9 +91,11 @@ struct btrfs_ordered_extent {*Thesefieldsdirectlycorrespondtothesamefieldsin*btrfs_file_extent_item.*/-u64disk_bytenr;u64num_bytes;+u64ram_bytes;+u64disk_bytenr;u64disk_num_bytes;+u64offset;/* number of bytes that still need writing */u64bytes_left;
From: Omar Sandoval <redacted>
Currently, we always reserve the same extent size in the file and extent
size on disk for delalloc because the former is the worst case for the
latter. For RWF_ENCODED writes, we know the exact size of the extent on
disk, which may be less than or greater than (for bookends) the size in
the file. Add a disk_num_bytes parameter to
btrfs_delalloc_reserve_metadata() so that we can reserve the correct
amount of csum bytes. No functional change.
Reviewed-by: Nikolay Borisov <redacted>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Omar Sandoval <redacted>
---
fs/btrfs/ctree.h | 3 ++-
fs/btrfs/delalloc-space.c | 18 ++++++++++--------
fs/btrfs/file.c | 3 ++-
fs/btrfs/inode.c | 2 +-
fs/btrfs/relocation.c | 2 +-
5 files changed, 16 insertions(+), 12 deletions(-)
@@ -3017,7 +3017,7 @@ static int relocate_one_page(struct inode *inode, struct file_ra_state *ra,/* Reserve metadata for this range */ret=btrfs_delalloc_reserve_metadata(BTRFS_I(inode),-clamped_len);+clamped_len,clamped_len);if(ret)gotorelease_page;
From: Omar Sandoval <redacted>
Currently, an inline extent is always created after i_size is extended
from btrfs_dirty_pages(). However, for encoded writes, we only want to
update i_size after we successfully created the inline extent. Add an
update_i_size parameter to cow_file_range_inline() and
insert_inline_extent() and pass in the size of the extent rather than
determining it from i_size. Since the start parameter is always passed
as 0, get rid of it and simplify the logic in these two functions. While
we're here, let's document the requirements for creating an inline
extent.
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Omar Sandoval <redacted>
---
fs/btrfs/inode.c | 100 +++++++++++++++++++++++------------------------
1 file changed, 48 insertions(+), 52 deletions(-)
@@ -695,14 +687,15 @@ static noinline int compress_file_range(struct async_chunk *async_chunk)/* we didn't compress the entire range, try*tomakeanuncompressedinlineextent.*/-ret=cow_file_range_inline(BTRFS_I(inode),start,end,+ret=cow_file_range_inline(BTRFS_I(inode),actual_end,0,BTRFS_COMPRESS_NONE,-NULL);+NULL,false);}else{/* try making a compressed inline extent */-ret=cow_file_range_inline(BTRFS_I(inode),start,end,+ret=cow_file_range_inline(BTRFS_I(inode),actual_end,total_compressed,-compress_type,pages);+compress_type,pages,+false);}if(ret<=0){unsignedlongclear_flags=EXTENT_DELALLOC|
@@ -1098,9 +1091,12 @@ static noinline int cow_file_range(struct btrfs_inode *inode,*Sohereweskipinlineextentcreationcompletely.*/if(start==0&&fs_info->sectorsize==PAGE_SIZE){+u64actual_end=min_t(u64,i_size_read(&inode->vfs_inode),+end+1);+/* lets try to make an inline extent */-ret=cow_file_range_inline(inode,start,end,0,-BTRFS_COMPRESS_NONE,NULL);+ret=cow_file_range_inline(inode,actual_end,0,+BTRFS_COMPRESS_NONE,NULL,false);if(ret==0){/**WeuseDO_ACCOUNTINGherebecauseweneedthe
From: Omar Sandoval <redacted>
In order to allow sending and receiving compressed data without
decompressing it, we need an interface to write pre-compressed data
directly to the filesystem and the matching interface to read compressed
data without decompressing it. This adds the definitions for ioctls to
do that and detailed explanations of how to use them.
Signed-off-by: Omar Sandoval <redacted>
---
include/uapi/linux/btrfs.h | 132 +++++++++++++++++++++++++++++++++++++
1 file changed, 132 insertions(+)
@@ -861,6 +861,134 @@ struct btrfs_ioctl_get_subvol_rootref_args {__u8align[7];};+/*+*Dataandmetadataforanencodedreadorwrite.+*+*EncodedI/Obypassesanyencodingautomaticallydonebythefilesystem(e.g.,+*compression).Thiscanbeusedtoreadthecompressedcontentsofafileor+*writepre-compresseddatadirectlytoafile.+*+*BTRFS_IOC_ENCODED_READandBTRFS_IOC_ENCODED_WRITEareessentially+*preadv/pwritevwithadditionalmetadataabouthowthedataisencodedandthe+*sizeoftheunencodeddata.+*+*BTRFS_IOC_ENCODED_READfillsthegiveniovecswiththeencodeddata,fills+*themetadatafields,andreturnsthesizeoftheencodeddata.Itreadsone+*extentpercall.Itcanalsoreaddatawhichisnotencoded.+*+*BTRFS_IOC_ENCODED_WRITEusesthemetadatafields,writestheencodeddata+*fromtheiovecs,andreturnsthesizeoftheencodeddata.Notethatthe+*encodeddataisnotvalidatedwhenitiswritten;ifitisnotvalid(e.g.,+*itcannotbedecompressed),thenasubsequentreadmayreturnanerror.+*+*Sincethefilesystempagecachecontainsdecodeddata,encodedI/Obypasses+*thepagecache.EncodedI/OrequiresCAP_SYS_ADMIN.+*/+structbtrfs_ioctl_encoded_io_args{+/* Input parameters for both reads and writes. */++/*+*iovecscontainingencodeddata.+*+*Forreads,ifthesizeoftheencodeddataislargerthanthesumof+*iov[n].iov_lenfor0<=n<iovcnt,thentheioctlfailswith+*ENOBUFS.+*+*Forwrites,thesizeoftheencodeddataisthesumofiov[n].iov_len+*for0<=n<iovcnt.Thismustbelessthan128KiB(thislimitmay+*increaseinthefuture).Thismustalsobelessthanorequalto+*unencoded_len.+*/+conststructiovec__user*iov;+/* Number of iovecs. */+unsignedlongiovcnt;+/*+*Offsetinfile.+*+*Forwrites,mustbealignedtothesectorsizeofthefilesystem.+*/+__s64offset;+/* Currently must be zero. */+__u64flags;++/*+*Forreads,thefollowingmembersarefilledinwiththemetadatafor+*theencodeddata.+*Forwrites,thefollowingmembersmustbesettothemetadataforthe+*encodeddata.+*/++/*+*Lengthofthedatainthefile.+*+*Mustbelessthanorequaltounencoded_len-unencoded_offset.For+*writes,mustbealignedtothesectorsizeofthefilesystemunless+*thedataendsatorbeyondthecurrentendofthefile.+*/+__u64len;+/*+*Lengthoftheunencoded(i.e.,decryptedanddecompressed)data.+*+*Forwrites,mustbenomorethan128KiB(thislimitmayincreasein+*thefuture).Iftheunencodeddataisactuallylongerthan+*unencoded_len,thenitistruncated;ifitisshorter,thenitis+*extendedwithzeroes.+*/+__u64unencoded_len;+/*+*Offsetfromthefirstbyteoftheunencodeddatatothefirstbyteof+*logicaldatainthefile.+*+*Mustbelessthanunencoded_len.+*/+__u64unencoded_offset;+/*+*BTRFS_ENCODED_IO_COMPRESSION_*type.+*+*Forwrites,mustnotbeBTRFS_ENCODED_IO_COMPRESSION_NONE.+*/+__u32compression;+/* Currently always BTRFS_ENCODED_IO_ENCRYPTION_NONE. */+__u32encryption;+/*+*Reservedforfutureexpansion.+*+*Forreads,alwaysreturnedaszero.Usersshouldcheckfornon-zero+*bytes.Ifthereareany,thenthekernelhasanewerversionofthis+*structurewithadditionalinformationthattheuserdefinitionis+*missing.+*+*Forwrites,mustbezeroed.+*/+__u8reserved[32];+};++/* Data is not compressed. */+#define BTRFS_ENCODED_IO_COMPRESSION_NONE 0+/* Data is compressed as a single zlib stream. */+#define BTRFS_ENCODED_IO_COMPRESSION_ZLIB 1+/*+*DataiscompressedasasinglezstdframewiththewindowLogcompression+*parametersettonomorethan17.+*/+#define BTRFS_ENCODED_IO_COMPRESSION_ZSTD 2+/*+*Dataiscompressedpagebypage(usingthepagesizeindicatedbythenameof+*theconstant)withLZO1Xandwrappedintheformatdocumentedin+*fs/btrfs/lzo.c.Forwrites,thecompressionpagesizemustmatchthe+*filesystempagesize.+*/+#define BTRFS_ENCODED_IO_COMPRESSION_LZO_4K 3+#define BTRFS_ENCODED_IO_COMPRESSION_LZO_8K 4+#define BTRFS_ENCODED_IO_COMPRESSION_LZO_16K 5+#define BTRFS_ENCODED_IO_COMPRESSION_LZO_32K 6+#define BTRFS_ENCODED_IO_COMPRESSION_LZO_64K 7+#define BTRFS_ENCODED_IO_COMPRESSION_TYPES 8++/* Data is not encrypted. */+#define BTRFS_ENCODED_IO_ENCRYPTION_NONE 0+#define BTRFS_ENCODED_IO_ENCRYPTION_TYPES 1+/* Error codes as returned by the kernel */enumbtrfs_err_code{BTRFS_ERROR_DEV_RAID1_MIN_NOT_MET=1,
From: Omar Sandoval <redacted>
There are 4 main cases:
1. Inline extents: we copy the data straight out of the extent buffer.
2. Hole/preallocated extents: we fill in zeroes.
3. Regular, uncompressed extents: we read the sectors we need directly
from disk.
4. Regular, compressed extents: we read the entire compressed extent
from disk and indicate what subset of the decompressed extent is in
the file.
This initial implementation simplifies a few things that can be improved
in the future:
- We hold the inode lock during the operation.
- Cases 1, 3, and 4 allocate temporary memory to read into before
copying out to userspace.
- We don't do read repair, because it turns out that read repair is
currently broken for compressed data.
Signed-off-by: Omar Sandoval <redacted>
---
fs/btrfs/ctree.h | 4 +
fs/btrfs/inode.c | 489 +++++++++++++++++++++++++++++++++++++++++++++++
fs/btrfs/ioctl.c | 111 +++++++++++
3 files changed, 604 insertions(+)
@@ -87,6 +88,22 @@ struct btrfs_ioctl_send_args_32 {#define BTRFS_IOC_SEND_32 _IOW(BTRFS_IOCTL_MAGIC, 38, \structbtrfs_ioctl_send_args_32)++structbtrfs_ioctl_encoded_io_args_32{+compat_uptr_tiov;+compat_ulong_tiovcnt;+__s64offset;+__u64flags;+__u64len;+__u64unencoded_len;+__u64unencoded_offset;+__u32compression;+__u32encryption;+__u32reserved[8];+};++#define BTRFS_IOC_ENCODED_READ_32 _IOR(BTRFS_IOCTL_MAGIC, 64, \+structbtrfs_ioctl_encoded_io_args_32)#endif/* Mask out flags that are inappropriate for the given type of inode. */
From: Omar Sandoval <redacted>
The implementation resembles direct I/O: we have to flush any ordered
extents, invalidate the page cache, and do the io tree/delalloc/extent
map/ordered extent dance. From there, we can reuse the compression code
with a minor modification to distinguish the write from writeback. This
also creates inline extents when possible.
Signed-off-by: Omar Sandoval <redacted>
---
fs/btrfs/compression.c | 7 +-
fs/btrfs/compression.h | 6 +-
fs/btrfs/ctree.h | 4 +
fs/btrfs/file.c | 65 ++++++++--
fs/btrfs/inode.c | 256 +++++++++++++++++++++++++++++++++++++++-
fs/btrfs/ioctl.c | 102 ++++++++++++++++
fs/btrfs/ordered-data.c | 12 +-
fs/btrfs/ordered-data.h | 5 +-
8 files changed, 437 insertions(+), 20 deletions(-)
@@ -353,7 +353,8 @@ static void end_compressed_bio_write(struct bio *bio)cb->start,cb->start+cb->len-1,!cb->errors);-end_compressed_writeback(inode,cb);+if(cb->writeback)+end_compressed_writeback(inode,cb);/* note, our inode could be gone now *//*
@@ -52,6 +52,9 @@ struct compressed_bio {/* The compression algorithm for this bio */u8compress_type;+/* Whether this is a write for writeback. */+boolwriteback;+/* IO errors */u8errors;intmirror_num;
@@ -2967,6 +2967,7 @@ static int insert_ordered_extent_file_extent(struct btrfs_trans_handle *trans,*exceptiftheorderedextentwastruncated.*/update_inode_bytes=test_bit(BTRFS_ORDERED_DIRECT,&oe->flags)||+test_bit(BTRFS_ORDERED_ENCODED,&oe->flags)||test_bit(BTRFS_ORDERED_TRUNCATED,&oe->flags);returninsert_reserved_file_extent(trans,BTRFS_I(oe->inode),
@@ -3001,7 +3002,8 @@ static int btrfs_finish_ordered_io(struct btrfs_ordered_extent *ordered_extent)if(!test_bit(BTRFS_ORDERED_NOCOW,&ordered_extent->flags)&&!test_bit(BTRFS_ORDERED_PREALLOC,&ordered_extent->flags)&&-!test_bit(BTRFS_ORDERED_DIRECT,&ordered_extent->flags))+!test_bit(BTRFS_ORDERED_DIRECT,&ordered_extent->flags)&&+!test_bit(BTRFS_ORDERED_ENCODED,&ordered_extent->flags))clear_bits|=EXTENT_DELALLOC_NEW;freespace_inode=btrfs_is_free_space_inode(inode);
@@ -10989,6 +10991,256 @@ ssize_t btrfs_encoded_read(struct kiocb *iocb, struct iov_iter *iter,returnret;}+ssize_tbtrfs_do_encoded_write(structkiocb*iocb,structiov_iter*from,+conststructbtrfs_ioctl_encoded_io_args*encoded)+{+structinode*inode=file_inode(iocb->ki_filp);+structbtrfs_fs_info*fs_info=btrfs_sb(inode->i_sb);+structbtrfs_root*root=BTRFS_I(inode)->root;+structextent_io_tree*io_tree=&BTRFS_I(inode)->io_tree;+structextent_changeset*data_reserved=NULL;+structextent_state*cached_state=NULL;+intcompression;+size_torig_count;+u64start,end;+u64num_bytes,ram_bytes,disk_num_bytes;+unsignedlongnr_pages,i;+structpage**pages;+structbtrfs_keyins;+boolextent_reserved=false;+structextent_map*em;+ssize_tret;++switch(encoded->compression){+caseBTRFS_ENCODED_IO_COMPRESSION_ZLIB:+compression=BTRFS_COMPRESS_ZLIB;+break;+caseBTRFS_ENCODED_IO_COMPRESSION_ZSTD:+compression=BTRFS_COMPRESS_ZSTD;+break;+caseBTRFS_ENCODED_IO_COMPRESSION_LZO_4K:+caseBTRFS_ENCODED_IO_COMPRESSION_LZO_8K:+caseBTRFS_ENCODED_IO_COMPRESSION_LZO_16K:+caseBTRFS_ENCODED_IO_COMPRESSION_LZO_32K:+caseBTRFS_ENCODED_IO_COMPRESSION_LZO_64K:+/* The page size must match for LZO. */+if(encoded->compression-+BTRFS_ENCODED_IO_COMPRESSION_LZO_4K+12!=PAGE_SHIFT)+return-EINVAL;+compression=BTRFS_COMPRESS_LZO;+break;+default:+return-EINVAL;+}+if(encoded->encryption!=BTRFS_ENCODED_IO_ENCRYPTION_NONE)+return-EINVAL;++orig_count=iov_iter_count(from);++/* The extent size must be sane. */+if(encoded->unencoded_len>BTRFS_MAX_UNCOMPRESSED||+orig_count>BTRFS_MAX_COMPRESSED||orig_count==0)+return-EINVAL;++/*+*Thecompresseddatamustbesmallerthanthedecompresseddata.+*+*It'sofcoursepossiblefordatatocompresstolargerorthesame+*size,butthebufferedI/Opathfallsbacktonocompressionforsuch+*data,andwedon'twanttobreakanyassumptionsbycreatingthese+*extents.+*+*Notethatthisislessstrictthanthecurrentcheckwehavethatthe+*compresseddatamustbeatleastonesectorsmallerthanthe+*decompresseddata.Weonlywanttoenforcetheweakerrequirement+*fromoldkernelsthatitisatleastonebytesmaller.+*/+if(orig_count>=encoded->unencoded_len)+return-EINVAL;++/* The extent must start on a sector boundary. */+start=iocb->ki_pos;+if(!IS_ALIGNED(start,fs_info->sectorsize))+return-EINVAL;++/*+*Theextentmustendonasectorboundary.However,weallowawrite+*whichendsatorextendsi_sizetohaveanunalignedlength;weround+*uptheextentsizeandseti_sizetotheunalignedend.+*/+if(start+encoded->len<inode->i_size&&+!IS_ALIGNED(start+encoded->len,fs_info->sectorsize))+return-EINVAL;++/* Finally, the offset in the unencoded data must be sector-aligned. */+if(!IS_ALIGNED(encoded->unencoded_offset,fs_info->sectorsize))+return-EINVAL;++num_bytes=ALIGN(encoded->len,fs_info->sectorsize);+ram_bytes=ALIGN(encoded->unencoded_len,fs_info->sectorsize);+end=start+num_bytes-1;++/*+*Iftheextentcannotbeinline,thecompresseddataondiskmustbe+*sector-aligned.Forconvenience,weextenditwithzeroesifit+*isn't.+*/+disk_num_bytes=ALIGN(orig_count,fs_info->sectorsize);+nr_pages=DIV_ROUND_UP(disk_num_bytes,PAGE_SIZE);+pages=kvcalloc(nr_pages,sizeof(structpage*),GFP_KERNEL_ACCOUNT);+if(!pages)+return-ENOMEM;+for(i=0;i<nr_pages;i++){+size_tbytes=min_t(size_t,PAGE_SIZE,iov_iter_count(from));+char*kaddr;++pages[i]=alloc_page(GFP_KERNEL_ACCOUNT|__GFP_HIGHMEM);+if(!pages[i]){+ret=-ENOMEM;+gotoout_pages;+}+kaddr=kmap(pages[i]);+if(copy_from_iter(kaddr,bytes,from)!=bytes){+kunmap(pages[i]);+ret=-EFAULT;+gotoout_pages;+}+if(bytes<PAGE_SIZE)+memset(kaddr+bytes,0,PAGE_SIZE-bytes);+kunmap(pages[i]);+}++for(;;){+structbtrfs_ordered_extent*ordered;++ret=btrfs_wait_ordered_range(inode,start,num_bytes);+if(ret)+gotoout_pages;+ret=invalidate_inode_pages2_range(inode->i_mapping,+start>>PAGE_SHIFT,+end>>PAGE_SHIFT);+if(ret)+gotoout_pages;+lock_extent_bits(io_tree,start,end,&cached_state);+ordered=btrfs_lookup_ordered_range(BTRFS_I(inode),start,+num_bytes);+if(!ordered&&+!filemap_range_has_page(inode->i_mapping,start,end))+break;+if(ordered)+btrfs_put_ordered_extent(ordered);+unlock_extent_cached(io_tree,start,end,&cached_state);+cond_resched();+}++/*+*Wedon'tusethehigher-leveldelallocspacefunctionsbecauseour+*num_bytesanddisk_num_bytesaredifferent.+*/+ret=btrfs_alloc_data_chunk_ondemand(BTRFS_I(inode),disk_num_bytes);+if(ret)+gotoout_unlock;+ret=btrfs_qgroup_reserve_data(BTRFS_I(inode),&data_reserved,start,+num_bytes);+if(ret)+gotoout_free_data_space;+ret=btrfs_delalloc_reserve_metadata(BTRFS_I(inode),num_bytes,+disk_num_bytes);+if(ret)+gotoout_qgroup_free_data;++/* Try an inline extent first. */+if(start==0&&encoded->unencoded_len==encoded->len&&+encoded->unencoded_offset==0){+ret=cow_file_range_inline(BTRFS_I(inode),encoded->len,+orig_count,compression,pages,+true);+if(ret<=0){+if(ret==0)+ret=orig_count;+gotoout_delalloc_release;+}+}++ret=btrfs_reserve_extent(root,disk_num_bytes,disk_num_bytes,+disk_num_bytes,0,0,&ins,1,1);+if(ret)+gotoout_delalloc_release;+extent_reserved=true;++em=create_io_em(BTRFS_I(inode),start,num_bytes,+start-encoded->unencoded_offset,ins.objectid,+ins.offset,ins.offset,ram_bytes,compression,+BTRFS_ORDERED_COMPRESSED);+if(IS_ERR(em)){+ret=PTR_ERR(em);+gotoout_free_reserved;+}+free_extent_map(em);++ret=btrfs_add_ordered_extent(BTRFS_I(inode),start,num_bytes,+ram_bytes,ins.objectid,ins.offset,+encoded->unencoded_offset,+(1<<BTRFS_ORDERED_ENCODED)|+(1<<BTRFS_ORDERED_COMPRESSED),+compression);+if(ret){+btrfs_drop_extent_cache(BTRFS_I(inode),start,end,0);+gotoout_free_reserved;+}+btrfs_dec_block_group_reservations(fs_info,ins.objectid);++if(start+encoded->len>inode->i_size)+i_size_write(inode,start+encoded->len);++unlock_extent_cached(io_tree,start,end,&cached_state);++btrfs_delalloc_release_extents(BTRFS_I(inode),num_bytes);++if(btrfs_submit_compressed_write(BTRFS_I(inode),start,num_bytes,+ins.objectid,ins.offset,pages,+nr_pages,0,NULL,false)){+btrfs_writepage_endio_finish_ordered(BTRFS_I(inode),pages[0],+start,end,0);+ret=-EIO;+gotoout_pages;+}+ret=orig_count;+gotoout;++out_free_reserved:+btrfs_dec_block_group_reservations(fs_info,ins.objectid);+btrfs_free_reserved_extent(fs_info,ins.objectid,ins.offset,1);+out_delalloc_release:+btrfs_delalloc_release_extents(BTRFS_I(inode),num_bytes);+btrfs_delalloc_release_metadata(BTRFS_I(inode),disk_num_bytes,+ret<0);+out_qgroup_free_data:+if(ret<0){+btrfs_qgroup_free_data(BTRFS_I(inode),data_reserved,start,+num_bytes);+}+out_free_data_space:+/*+*Ifbtrfs_reserve_extent()succeeded,thenwealreadydecremented+*bytes_may_use.+*/+if(!extent_reserved)+btrfs_free_reserved_data_space_noquota(fs_info,disk_num_bytes);+out_unlock:+unlock_extent_cached(io_tree,start,end,&cached_state);+out_pages:+for(i=0;i<nr_pages;i++){+if(pages[i])+__free_page(pages[i]);+}+kvfree(pages);+out:+if(ret>=0)+iocb->ki_pos+=encoded->len;+returnret;+}+#ifdef CONFIG_SWAP/**Addanentryindicatingablockgroupordevicewhichispinnedbya
@@ -104,6 +104,8 @@ struct btrfs_ioctl_encoded_io_args_32 {#define BTRFS_IOC_ENCODED_READ_32 _IOR(BTRFS_IOCTL_MAGIC, 64, \structbtrfs_ioctl_encoded_io_args_32)+#define BTRFS_IOC_ENCODED_WRITE_32 _IOW(BTRFS_IOCTL_MAGIC, 64, \+structbtrfs_ioctl_encoded_io_args_32)#endif/* Mask out flags that are inappropriate for the given type of inode. */
@@ -74,6 +74,8 @@ enum {BTRFS_ORDERED_LOGGED_CSUM,/* We wait for this extent to complete in the current transaction */BTRFS_ORDERED_PENDING,+/* BTRFS_IOC_ENCODED_WRITE */+BTRFS_ORDERED_ENCODED,};/* BTRFS_ORDERED_* flags that specify the type of the extent. */
@@ -81,7 +83,8 @@ enum {(1UL<<BTRFS_ORDERED_NOCOW)|\(1UL<<BTRFS_ORDERED_PREALLOC)|\(1UL<<BTRFS_ORDERED_COMPRESSED)|\-(1UL<<BTRFS_ORDERED_DIRECT))+(1UL<<BTRFS_ORDERED_DIRECT)|\+(1UL<<BTRFS_ORDERED_ENCODED))structbtrfs_ordered_extent{/* logical offset in the file */
From: Omar Sandoval <redacted>
The length field of the send stream TLV header is 16 bits. This means
that the maximum amount of data that can be sent for one write is 64k
minus one. However, encoded writes must be able to send the maximum
compressed extent (128k) in one command. To support this, send stream
version 2 encodes the DATA attribute differently: it has no length
field, and the length is implicitly up to the end of containing command
(which has a 32-bit length field). Although this is necessary for
encoded writes, normal writes can benefit from it, too.
For v2, let's bump up the send buffer to the maximum compressed extent
size plus 16k for the other metadata (144k total). Since this will most
likely be vmalloc'd (and always will be after the next commit), we round
it up to the next page since we might as well use the rest of the page
on systems with >16k pages.
Signed-off-by: Omar Sandoval <redacted>
---
fs/btrfs/send.c | 34 ++++++++++++++++++++++++++--------
1 file changed, 26 insertions(+), 8 deletions(-)
From: Omar Sandoval <redacted>
This adds the definitions of the new commands for send stream version 2
and their respective attributes: fallocate, FS_IOC_SETFLAGS (a.k.a.
chattr), and encoded writes. It also documents two changes to the send
stream format in v2: the receiver shouldn't assume a maximum command
size, and the DATA attribute is encoded differently to allow for writes
larger than 64k. These will be implemented in subsequent changes, and
then the ioctl will accept the new flags.
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Omar Sandoval <redacted>
---
fs/btrfs/send.c | 2 +-
fs/btrfs/send.h | 30 +++++++++++++++++++++++++++++-
include/uapi/linux/btrfs.h | 13 +++++++++++++
3 files changed, 43 insertions(+), 2 deletions(-)
@@ -114,6 +130,18 @@ enum {BTRFS_SEND_A_CLONE_OFFSET,BTRFS_SEND_A_CLONE_LEN,+/* The following attributes were added in send stream v2. */++BTRFS_SEND_A_FALLOCATE_MODE,++BTRFS_SEND_A_SETFLAGS_FLAGS,++BTRFS_SEND_A_UNENCODED_FILE_LEN,+BTRFS_SEND_A_UNENCODED_LEN,+BTRFS_SEND_A_UNENCODED_OFFSET,+BTRFS_SEND_A_COMPRESSION,+BTRFS_SEND_A_ENCRYPTION,+__BTRFS_SEND_A_MAX,};#define BTRFS_SEND_A_MAX (__BTRFS_SEND_A_MAX - 1)
From: Omar Sandoval <redacted>
For encoded writes, we need the raw pages for reading compressed data
directly via a bio. So, replace kvmalloc() with vmap() so we have access
to the raw pages. 144k is large enough that it usually gets allocated
with vmalloc(), anyways.
Signed-off-by: Omar Sandoval <redacted>
---
fs/btrfs/send.c | 33 +++++++++++++++++++++++++++++++--
1 file changed, 31 insertions(+), 2 deletions(-)
@@ -81,6 +81,7 @@ struct send_ctx {char*send_buf;u32send_size;u32send_max_size;+structpage**send_buf_pages;u64total_send_size;u64cmd_send_size[BTRFS_SEND_C_MAX+1];u64flags;/* 'flags' member of btrfs_ioctl_send_args is u64 */
From: Omar Sandoval <redacted>
Now that all of the pieces are in place, we can use the ENCODED_WRITE
command to send compressed extents when appropriate.
Signed-off-by: Omar Sandoval <redacted>
---
fs/btrfs/ctree.h | 4 +
fs/btrfs/inode.c | 6 +-
fs/btrfs/send.c | 230 +++++++++++++++++++++++++++++++++++++++++++----
3 files changed, 220 insertions(+), 20 deletions(-)
From: Omar Sandoval <redacted>
Now that the new support is implemented, allow the ioctl to accept the
flags and update the version in sysfs.
Signed-off-by: Omar Sandoval <redacted>
---
fs/btrfs/send.c | 10 +++++++++-
fs/btrfs/send.h | 2 +-
include/uapi/linux/btrfs.h | 4 +++-
3 files changed, 13 insertions(+), 3 deletions(-)
From: Boris Burkov <redacted>
An encoded extent can be up to 128K in length, which exceeds the largest
value expressible by the current send stream format's 16 bit tlv_len
field. Since encoded writes cannot be split into multiple writes by
btrfs send, the send stream format must change to accommodate encoded
writes.
Supporting this changed format requires retooling how we store the
commands we have processed. Since we can no longer use btrfs_tlv_header
to describe every attribute, we define a new struct btrfs_send_attribute
which has a 32 bit length field, and use that to store the attribute
information needed for receive processing. This is transparent to users
of the various TLV_GET macros.
Signed-off-by: Boris Burkov <redacted>
---
common/send-stream.c | 34 +++++++++++++++++++++++++---------
1 file changed, 25 insertions(+), 9 deletions(-)
@@ -188,15 +204,15 @@ static int tlv_get(struct btrfs_send_stream *sctx, int attr, void **data, int *lgotoout;}-hdr=sctx->cmd_attrs[attr];-if(!hdr){+send_attr=&sctx->cmd_attrs[attr];+if(!send_attr->data){error("attribute %d requested but not present",attr);ret=-ENOENT;gotoout;}-*len=le16_to_cpu(hdr->tlv_len);-*data=hdr+1;+*len=send_attr->tlv_len;+*data=send_attr->data;ret=0;
From: Boris Burkov <redacted>
In send stream v2, write commands can now be an arbitrary size. For that
reason, we can no longer allocate a fixed array in sctx for read_cmd.
Instead, read_cmd dynamically allocates sctx->read_buf. To avoid
needless reallocations, we reuse read_buf between read_cmd calls by also
keeping track of the size of the allocated buffer in sctx->read_buf_sz.
We do the first allocation of the old default size at the start of
processing the stream, and we only reallocate if we encounter a command
that needs a larger buffer.
Signed-off-by: Boris Burkov <redacted>
---
common/send-stream.c | 55 ++++++++++++++++++++++++++++----------------
send.h | 2 +-
2 files changed, 36 insertions(+), 21 deletions(-)
@@ -111,11 +111,12 @@ static int read_cmd(struct btrfs_send_stream *sctx)u32pos;u32crc;u32crc2;+structbtrfs_cmd_header*cmd_hdr;+size_tbuf_len;memset(sctx->cmd_attrs,0,sizeof(sctx->cmd_attrs));-ASSERT(sizeof(*sctx->cmd_hdr)<=sizeof(sctx->read_buf));-ret=read_buf(sctx,sctx->read_buf,sizeof(*sctx->cmd_hdr));+ret=read_buf(sctx,sctx->read_buf,sizeof(*cmd_hdr));if(ret<0)gotoout;if(ret){
@@ -124,18 +125,22 @@ static int read_cmd(struct btrfs_send_stream *sctx)gotoout;}-sctx->cmd_hdr=(structbtrfs_cmd_header*)sctx->read_buf;-cmd=le16_to_cpu(sctx->cmd_hdr->cmd);-cmd_len=le32_to_cpu(sctx->cmd_hdr->len);--if(cmd_len+sizeof(*sctx->cmd_hdr)>=sizeof(sctx->read_buf)){-ret=-EINVAL;-error("command length %u too big for buffer %zu",-cmd_len,sizeof(sctx->read_buf));-gotoout;+cmd_hdr=(structbtrfs_cmd_header*)sctx->read_buf;+cmd_len=le32_to_cpu(cmd_hdr->len);+cmd=le16_to_cpu(cmd_hdr->cmd);+buf_len=sizeof(*cmd_hdr)+cmd_len;+if(sctx->read_buf_sz<buf_len){+sctx->read_buf=realloc(sctx->read_buf,buf_len);+if(!sctx->read_buf){+ret=-ENOMEM;+error("failed to reallocate read buffer for cmd");+gotoout;+}+sctx->read_buf_sz=buf_len;+/* We need to reset cmd_hdr after realloc of sctx->read_buf */+cmd_hdr=(structbtrfs_cmd_header*)sctx->read_buf;}--data=sctx->read_buf+sizeof(*sctx->cmd_hdr);+data=sctx->read_buf+sizeof(*cmd_hdr);ret=read_buf(sctx,data,cmd_len);if(ret<0)gotoout;
@@ -145,11 +150,12 @@ static int read_cmd(struct btrfs_send_stream *sctx)gotoout;}-crc=le32_to_cpu(sctx->cmd_hdr->crc);-sctx->cmd_hdr->crc=0;+crc=le32_to_cpu(cmd_hdr->crc);+/* in send, crc is computed with header crc = 0, replicate that */+cmd_hdr->crc=0;crc2=crc32c(0,(unsignedchar*)sctx->read_buf,-sizeof(*sctx->cmd_hdr)+cmd_len);+sizeof(*cmd_hdr)+cmd_len);if(crc!=crc2){ret=-EINVAL;
@@ -524,19 +530,28 @@ int btrfs_read_and_process_send_stream(int fd,gotoout;}+sctx.read_buf=malloc(BTRFS_SEND_BUF_SIZE_V1);+if(!sctx.read_buf){+ret=-ENOMEM;+error("unable to allocate send stream read buffer");+gotoout;+}+sctx.read_buf_sz=BTRFS_SEND_BUF_SIZE_V1;+while(1){ret=read_and_process_cmd(&sctx);if(ret<0){last_err=ret;errors++;if(max_errors>0&&errors>=max_errors)-gotoout;+break;}elseif(ret>0){if(!honor_end_cmd)ret=0;-gotoout;+break;}}+free(sctx.read_buf);out:if(last_err&&!ret)
From: Boris Burkov <redacted>
The new format privileges the BTRFS_SEND_A_DATA attribute by
guaranteeing it will always be the last attribute in any command that
needs it, and by implicitly encoding the data length as the difference
between the total command length in the command header and the sizes of
the rest of the attributes (and of course the tlv_type identifying the
DATA attribute). To parse the new stream, we must read the tlv_type and
if it is not DATA, we proceed normally, but if it is DATA, we don't
parse a tlv_len but simply compute the length.
In addition, we add some bounds checking when parsing each chunk of
data, as well as for the tlv_len itself.
Signed-off-by: Boris Burkov <redacted>
---
common/send-stream.c | 36 ++++++++++++++++++++++++++----------
1 file changed, 26 insertions(+), 10 deletions(-)
From: Boris Burkov <redacted>
Send stream v2 adds three commands and several attributes associated to
those commands. Before we implement processing them, add all the
commands and attributes. This avoids leaving the enums in an
intermediate state that doesn't correspond to any version of send
stream.
Signed-off-by: Boris Burkov <redacted>
---
send.h | 15 +++++++++++++++
1 file changed, 15 insertions(+)
From: Boris Burkov <redacted>
Add a new btrfs_send_op and support for both dumping and proper receive
processing which does actual encoded writes.
Encoded writes are only allowed on a file descriptor opened with an
extra flag that allows encoded writes, so we also add support for this
flag when opening or reusing a file for writing.
Signed-off-by: Boris Burkov <redacted>
---
cmds/receive-dump.c | 16 +++++-
cmds/receive.c | 47 +++++++++++++++
common/send-stream.c | 22 ++++++++
common/send-stream.h | 4 ++
ioctl.h | 132 +++++++++++++++++++++++++++++++++++++++++++
5 files changed, 220 insertions(+), 1 deletion(-)
@@ -775,6 +775,134 @@ struct btrfs_ioctl_get_subvol_rootref_args {};BUILD_ASSERT(sizeof(structbtrfs_ioctl_get_subvol_rootref_args)==4096);+/*+*Dataandmetadataforanencodedreadorwrite.+*+*EncodedI/Obypassesanyencodingautomaticallydonebythefilesystem(e.g.,+*compression).Thiscanbeusedtoreadthecompressedcontentsofafileor+*writepre-compresseddatadirectlytoafile.+*+*BTRFS_IOC_ENCODED_READandBTRFS_IOC_ENCODED_WRITEareessentially+*preadv/pwritevwithadditionalmetadataabouthowthedataisencodedandthe+*sizeoftheunencodeddata.+*+*BTRFS_IOC_ENCODED_READfillsthegiveniovecswiththeencodeddata,fills+*themetadatafields,andreturnsthesizeoftheencodeddata.Itreadsone+*extentpercall.Itcanalsoreaddatawhichisnotencoded.+*+*BTRFS_IOC_ENCODED_WRITEusesthemetadatafields,writestheencodeddata+*fromtheiovecs,andreturnsthesizeoftheencodeddata.Notethatthe+*encodeddataisnotvalidatedwhenitiswritten;ifitisnotvalid(e.g.,+*itcannotbedecompressed),thenasubsequentreadmayreturnanerror.+*+*Sincethefilesystempagecachecontainsdecodeddata,encodedI/Obypasses+*thepagecache.EncodedI/OrequiresCAP_SYS_ADMIN.+*/+structbtrfs_ioctl_encoded_io_args{+/* Input parameters for both reads and writes. */++/*+*iovecscontainingencodeddata.+*+*Forreads,ifthesizeoftheencodeddataislargerthanthesumof+*iov[n].iov_lenfor0<=n<iovcnt,thentheioctlfailswith+*ENOBUFS.+*+*Forwrites,thesizeoftheencodeddataisthesumofiov[n].iov_len+*for0<=n<iovcnt.Thismustbelessthan128KiB(thislimitmay+*increaseinthefuture).Thismustalsobelessthanorequalto+*unencoded_len.+*/+conststructiovec*iov;+/* Number of iovecs. */+unsignedlongiovcnt;+/*+*Offsetinfile.+*+*Forwrites,mustbealignedtothesectorsizeofthefilesystem.+*/+__s64offset;+/* Currently must be zero. */+__u64flags;++/*+*Forreads,thefollowingmembersarefilledinwiththemetadatafor+*theencodeddata.+*Forwrites,thefollowingmembersmustbesettothemetadataforthe+*encodeddata.+*/++/*+*Lengthofthedatainthefile.+*+*Mustbelessthanorequaltounencoded_len-unencoded_offset.For+*writes,mustbealignedtothesectorsizeofthefilesystemunless+*thedataendsatorbeyondthecurrentendofthefile.+*/+__u64len;+/*+*Lengthoftheunencoded(i.e.,decryptedanddecompressed)data.+*+*Forwrites,mustbenomorethan128KiB(thislimitmayincreasein+*thefuture).Iftheunencodeddataisactuallylongerthan+*unencoded_len,thenitistruncated;ifitisshorter,thenitis+*extendedwithzeroes.+*/+__u64unencoded_len;+/*+*Offsetfromthefirstbyteoftheunencodeddatatothefirstbyteof+*logicaldatainthefile.+*+*Mustbelessthanunencoded_len.+*/+__u64unencoded_offset;+/*+*BTRFS_ENCODED_IO_COMPRESSION_*type.+*+*Forwrites,mustnotbeBTRFS_ENCODED_IO_COMPRESSION_NONE.+*/+__u32compression;+/* Currently always BTRFS_ENCODED_IO_ENCRYPTION_NONE. */+__u32encryption;+/*+*Reservedforfutureexpansion.+*+*Forreads,alwaysreturnedaszero.Usersshouldcheckfornon-zero+*bytes.Ifthereareany,thenthekernelhasanewerversionofthis+*structurewithadditionalinformationthattheuserdefinitionis+*missing.+*+*Forwrites,mustbezeroed.+*/+__u8reserved[32];+};++/* Data is not compressed. */+#define BTRFS_ENCODED_IO_COMPRESSION_NONE 0+/* Data is compressed as a single zlib stream. */+#define BTRFS_ENCODED_IO_COMPRESSION_ZLIB 1+/*+*DataiscompressedasasinglezstdframewiththewindowLogcompression+*parametersettonomorethan17.+*/+#define BTRFS_ENCODED_IO_COMPRESSION_ZSTD 2+/*+*Dataiscompressedpagebypage(usingthepagesizeindicatedbythenameof+*theconstant)withLZO1Xandwrappedintheformatdocumentedin+*fs/btrfs/lzo.c.Forwrites,thecompressionpagesizemustmatchthe+*filesystempagesize.+*/+#define BTRFS_ENCODED_IO_COMPRESSION_LZO_4K 3+#define BTRFS_ENCODED_IO_COMPRESSION_LZO_8K 4+#define BTRFS_ENCODED_IO_COMPRESSION_LZO_16K 5+#define BTRFS_ENCODED_IO_COMPRESSION_LZO_32K 6+#define BTRFS_ENCODED_IO_COMPRESSION_LZO_64K 7+#define BTRFS_ENCODED_IO_COMPRESSION_TYPES 8++/* Data is not encrypted. */+#define BTRFS_ENCODED_IO_ENCRYPTION_NONE 0+#define BTRFS_ENCODED_IO_ENCRYPTION_TYPES 1+/* Error codes as returned by the kernel */enumbtrfs_err_code{notused,
From: Boris Burkov <redacted>
Send stream v2 can emit fallocate commands, so receive must support them
as well. The implementation simply passes along the arguments to the
syscall. Note that mode is encoded as a u32 in send stream but fallocate
takes an int, so there is a unsigned->signed conversion there.
Signed-off-by: Boris Burkov <redacted>
---
cmds/receive-dump.c | 9 +++++++++
cmds/receive.c | 25 +++++++++++++++++++++++++
common/send-stream.c | 9 +++++++++
common/send-stream.h | 2 ++
4 files changed, 45 insertions(+)
From: Boris Burkov <redacted>
An encoded_write can fail if the file system it is being applied to does
not support encoded writes or if it can't find enough contiguous space
to accommodate the encoded extent. In those cases, we can likely still
process an encoded_write by explicitly decoding the data and doing a
normal write.
Add the necessary fallback path for decoding data compressed with zlib,
lzo, or zstd. zlib and zstd have reusable decoding context data
structures which we cache in the receive context so that we don't have
to recreate them on every encoded_write.
Finally, add a command line flag for force-decompress which causes
receive to always use the fallback path rather than first attempting the
encoded write.
Signed-off-by: Boris Burkov <redacted>
---
Documentation/btrfs-receive.asciidoc | 4 +
cmds/receive.c | 266 ++++++++++++++++++++++++++-
2 files changed, 261 insertions(+), 9 deletions(-)
@@ -60,6 +60,10 @@ By default the mountpoint is searched in '/proc/self/mounts'. If '/proc' is not accessible, eg. in a chroot environment, use this option to tell us where this filesystem is mounted.+--force-decompress::+if the stream contains compressed data (see '--compressed-data' in+`btrfs-send`(8)), always decompress it instead of writing it with encoded I/O.+ --dump:: dump the stream metadata, one line per operation +
@@ -989,9 +999,222 @@ static int process_update_extent(const char *path, u64 offset, u64 len,return0;}+staticintdecompress_zlib(structbtrfs_receive*rctx,constchar*encoded_data,+u64encoded_len,char*unencoded_data,+u64unencoded_len)+{+boolinit=false;+intret;++if(!rctx->zlib_stream){+init=true;+rctx->zlib_stream=malloc(sizeof(z_stream));+if(!rctx->zlib_stream){+error("failed to allocate zlib stream %m");+return-ENOMEM;+}+}+rctx->zlib_stream->next_in=(void*)encoded_data;+rctx->zlib_stream->avail_in=encoded_len;+rctx->zlib_stream->next_out=(void*)unencoded_data;+rctx->zlib_stream->avail_out=unencoded_len;++if(init){+rctx->zlib_stream->zalloc=Z_NULL;+rctx->zlib_stream->zfree=Z_NULL;+rctx->zlib_stream->opaque=Z_NULL;+ret=inflateInit(rctx->zlib_stream);+}else{+ret=inflateReset(rctx->zlib_stream);+}+if(ret!=Z_OK){+error("zlib inflate init failed: %d",ret);+return-EIO;+}++while(rctx->zlib_stream->avail_in>0&&+rctx->zlib_stream->avail_out>0){+ret=inflate(rctx->zlib_stream,Z_FINISH);+if(ret==Z_STREAM_END){+break;+}elseif(ret!=Z_OK){+error("zlib inflate failed: %d",ret);+return-EIO;+}+}+return0;+}++staticintdecompress_zstd(structbtrfs_receive*rctx,constchar*encoded_buf,+u64encoded_len,char*unencoded_buf,+u64unencoded_len)+{+ZSTD_inBufferin_buf={+.src=encoded_buf,+.size=encoded_len+};+ZSTD_outBufferout_buf={+.dst=unencoded_buf,+.size=unencoded_len+};+size_tret;++if(!rctx->zstd_dstream){+rctx->zstd_dstream=ZSTD_createDStream();+if(!rctx->zstd_dstream){+error("failed to create zstd dstream");+return-ENOMEM;+}+}+ret=ZSTD_initDStream(rctx->zstd_dstream);+if(ZSTD_isError(ret)){+error("failed to init zstd stream: %s",ZSTD_getErrorName(ret));+return-EIO;+}+while(in_buf.pos<in_buf.size&&out_buf.pos<out_buf.size){+ret=ZSTD_decompressStream(rctx->zstd_dstream,&out_buf,&in_buf);+if(ret==0){+break;+}elseif(ZSTD_isError(ret)){+error("failed to decompress zstd stream: %s",+ZSTD_getErrorName(ret));+return-EIO;+}+}+return0;+}++staticintdecompress_lzo(constchar*encoded_data,u64encoded_len,+char*unencoded_data,u64unencoded_len,+unsignedintpage_size)+{+uint32_ttotal_len;+size_tin_pos,out_pos;++if(encoded_len<4){+error("lzo header is truncated");+return-EIO;+}+memcpy(&total_len,encoded_data,4);+total_len=le32toh(total_len);+if(total_len>encoded_len){+error("lzo header is invalid");+return-EIO;+}++in_pos=4;+out_pos=0;+while(in_pos<total_len&&out_pos<unencoded_len){+size_tpage_remaining;+uint32_tsrc_len;+lzo_uintdst_len;+intret;++page_remaining=-in_pos%page_size;+if(page_remaining<4){+if(total_len-in_pos<=page_remaining)+break;+in_pos+=page_remaining;+}++if(total_len-in_pos<4){+error("lzo segment header is truncated");+return-EIO;+}++memcpy(&src_len,encoded_data+in_pos,4);+src_len=le32toh(src_len);+in_pos+=4;+if(src_len>total_len-in_pos){+error("lzo segment header is invalid");+return-EIO;+}++dst_len=page_size;+ret=lzo1x_decompress_safe((void*)(encoded_data+in_pos),+src_len,+(void*)(unencoded_data+out_pos),+&dst_len,NULL);+if(ret!=LZO_E_OK){+error("lzo1x_decompress_safe failed: %d",ret);+return-EIO;+}++in_pos+=src_len;+out_pos+=dst_len;+}+return0;+}++staticintdecompress_and_write(structbtrfs_receive*rctx,+constchar*encoded_data,u64offset,+u64encoded_len,u64unencoded_file_len,+u64unencoded_len,u64unencoded_offset,+u32compression)+{+intret=0;+size_tpos;+ssize_tw;+char*unencoded_data;+intpage_shift;++unencoded_data=calloc(unencoded_len,1);+if(!unencoded_data){+error("allocating space for unencoded data failed: %m");+return-errno;+}++switch(compression){+caseBTRFS_ENCODED_IO_COMPRESSION_ZLIB:+ret=decompress_zlib(rctx,encoded_data,encoded_len,+unencoded_data,unencoded_len);+if(ret)+gotoout;+break;+caseBTRFS_ENCODED_IO_COMPRESSION_ZSTD:+ret=decompress_zstd(rctx,encoded_data,encoded_len,+unencoded_data,unencoded_len);+if(ret)+gotoout;+break;+caseBTRFS_ENCODED_IO_COMPRESSION_LZO_4K:+caseBTRFS_ENCODED_IO_COMPRESSION_LZO_8K:+caseBTRFS_ENCODED_IO_COMPRESSION_LZO_16K:+caseBTRFS_ENCODED_IO_COMPRESSION_LZO_32K:+caseBTRFS_ENCODED_IO_COMPRESSION_LZO_64K:+page_shift=compression-BTRFS_ENCODED_IO_COMPRESSION_LZO_4K+12;+ret=decompress_lzo(encoded_data,encoded_len,unencoded_data,+unencoded_len,1U<<page_shift);+if(ret)+gotoout;+break;+default:+error("unknown compression: %d",compression);+ret=-EOPNOTSUPP;+gotoout;+}++pos=unencoded_offset;+while(pos<unencoded_file_len){+w=pwrite(rctx->write_fd,unencoded_data+pos,+unencoded_file_len-pos,offset);+if(w<0){+ret=-errno;+error("writing unencoded data failed: %m");+gotoout;+}+pos+=w;+offset+=w;+}+out:+free(unencoded_data);+returnret;+}+staticintprocess_encoded_write(constchar*path,constvoid*data,u64offset,-u64len,u64unencoded_file_len,u64unencoded_len,-u64unencoded_offset,u32compression,u32encryption,void*user)+u64len,u64unencoded_file_len,+u64unencoded_len,u64unencoded_offset,+u32compression,u32encryption,void*user){intret;structbtrfs_receive*rctx=user;
@@ -1007,6 +1230,7 @@ static int process_encoded_write(const char *path, const void *data, u64 offset,.compression=compression,.encryption=encryption,};+boolencoded_write=!rctx->force_decompress;if(encryption){error("encoded_write: encryption not supported");
@@ -1023,13 +1247,21 @@ static int process_encoded_write(const char *path, const void *data, u64 offset,if(ret<0)returnret;-ret=ioctl(rctx->write_fd,BTRFS_IOC_ENCODED_WRITE,&encoded);-if(ret<0){-ret=-errno;-error("encoded_write: writing to %s failed: %m",path);-returnret;+if(encoded_write){+ret=ioctl(rctx->write_fd,BTRFS_IOC_ENCODED_WRITE,&encoded);+if(ret>=0)+return0;+/* Fall back for these errors, fail hard for anything else. */+if(errno!=ENOSPC&&errno!=ENOTTY&&errno!=EINVAL){+ret=-errno;+error("encoded_write: writing to %s failed: %m",path);+returnret;+}}-return0;++returndecompress_and_write(rctx,data,offset,len,unencoded_file_len,+unencoded_len,unencoded_offset,+compression);}staticstructbtrfs_send_opssend_ops={
@@ -1239,6 +1477,9 @@ static const char * const cmd_receive_usage[] = {"-m ROOTMOUNT the root mount point of the destination filesystem."," If /proc is not accessible, use this to tell us where"," this file system is mounted.",+"--force-decompress",+" if the stream contains compressed data, always",+" decompress it instead of writing it with encoded I/O","--dump dump stream metadata, one line per operation,"," does not require the MOUNT parameter","-v deprecated, alias for global -v option",
@@ -1282,12 +1523,16 @@ static int cmd_receive(const struct cmd_struct *cmd, int argc, char **argv)optind=0;while(1){intc;-enum{GETOPT_VAL_DUMP=257};+enum{+GETOPT_VAL_DUMP=257,+GETOPT_VAL_FORCE_DECOMPRESS,+};staticconststructoptionlong_opts[]={{"max-errors",required_argument,NULL,'E'},{"chroot",no_argument,NULL,'C'},{"dump",no_argument,NULL,GETOPT_VAL_DUMP},{"quiet",no_argument,NULL,'q'},+{"force-decompress",no_argument,NULL,GETOPT_VAL_FORCE_DECOMPRESS},{NULL,0,NULL,0}};
@@ -1330,6 +1575,9 @@ static int cmd_receive(const struct cmd_struct *cmd, int argc, char **argv)caseGETOPT_VAL_DUMP:dump=1;break;+caseGETOPT_VAL_FORCE_DECOMPRESS:+rctx.force_decompress=1;+break;default:usage_unknown_option(cmd,argv);}
From: Boris Burkov <redacted>
In send stream v2, send can emit a command for setting inode flags via
the setflags ioctl. Pass the flags attribute through to the ioctl call
in receive.
Signed-off-by: Boris Burkov <redacted>
---
cmds/receive-dump.c | 6 ++++++
cmds/receive.c | 25 +++++++++++++++++++++++++
common/send-stream.c | 7 +++++++
common/send-stream.h | 1 +
4 files changed, 39 insertions(+)
From: Boris Burkov <redacted>
To make the btrfs send ioctl use the stream v2 format requires passing
BTRFS_SEND_FLAG_STREAM_V2 in flags. Further, to cause the ioctl to emit
encoded_write commands for encoded extents, we must set that flag as
well as BTRFS_SEND_FLAG_COMPRESSED. Finally, we bump up the version in
send.h as well, since we are now fully compatible with v2.
Add two command line arguments to btrfs send: --stream-version and
--compressed-data. --stream-version requires an argument which it parses
as an integer and sets STREAM_V2 if the argument is 2. --compressed-data
does not require an argument and automatically implies STREAM_V2 as well
(COMPRESSED alone causes the ioctl to error out).
Some examples to illustrate edge cases:
// v1, old format and no encoded_writes
btrfs send subvol
btrfs send --stream-version 1 subvol
// v2 and compressed, we will see encoded_writes
btrfs send --compressed-data subvol
btrfs send --compressed-data --stream-version 2 subvol
// v2 only, new format but no encoded_writes
btrfs send --stream-version 2 subvol
// error: compressed needs version >= 2
btrfs send --compressed-data --stream-version 1 subvol
// error: invalid version (not 1 or 2)
btrfs send --stream-version 3 subvol
btrfs send --compressed-data --stream-version 0 subvol
btrfs send --compressed-data --stream-version 10 subvol
Signed-off-by: Boris Burkov <redacted>
---
Documentation/btrfs-send.asciidoc | 16 ++++++++-
cmds/send.c | 54 ++++++++++++++++++++++++++++++-
ioctl.h | 17 +++++++++-
libbtrfsutil/btrfs.h | 17 +++++++++-
send.h | 2 +-
5 files changed, 101 insertions(+), 5 deletions(-)
@@ -55,7 +55,21 @@ send in 'NO_FILE_DATA' mode The output stream does not contain any file data and thus cannot be used to transfer changes. This mode is faster and is useful to show the differences in metadata.--q|--quiet::::++--stream-version <1|2>::+Use the given send stream version. The default is 1. Version 2 encodes file+data slightly more efficiently; it is also required for sending compressed data+directly (see '--compressed-data'). Version 2 requires at least btrfs-progs+5.12 on both the sender and receiver and at least Linux 5.12 on the sender.++--compressed-data::+Send data that is compressed on the filesystem directly without decompressing+it. If the receiver supports encoded I/O (see `encoded_io`(7)), it can also+write it directly without decompressing it. Otherwise, the receiver will fall+back to decompressing it and writing it normally. This implies+'--stream-version 2'.++-q|--quiet:: (deprecated) alias for global '-q' option -v|--verbose:: (deprecated) alias for global '-v' option
@@ -452,6 +452,21 @@ static const char * const cmd_send_usage[] = {" does not contain any file data and thus cannot be used"," to transfer changes. This mode is faster and useful to"," show the differences in metadata.",+"--stream-version <1|2>",+" Use the given send stream version. The default is",+" 1. Version 2 encodes file data slightly more",+" efficiently; it is also required for sending",+" compressed data directly (see --compressed-data).",+" Version 2 requires at least btrfs-progs 5.12 on both",+" the sender and receiver and at least Linux 5.12 on the",+" sender.",+"--compressed-data",+" Send data that is compressed on the filesystem",+" directly without decompressing it. If the receiver",+" supports encoded I/O, it can also write it directly",+" without decompressing it. Otherwise, the receiver will",+" fall back to decompressing it and writing it normally.",+" This implies --stream-version 2.","-v|--verbose deprecated, alias for global -v option","-q|--quiet deprecated, alias for global -q option",HELPINFO_INSERT_GLOBALS,
From: Boris Burkov <redacted>
Adapt the existing send/receive tests by passing '-o --force-compress'
to the mount commands in a new test. After writing a few files in the
various compression formats, send/receive them with and without
--force-decompress to test both the encoded_write path and the
fallback to decode+write.
Signed-off-by: Boris Burkov <redacted>
---
.../049-receive-write-encoded/test.sh | 114 ++++++++++++++++++
1 file changed, 114 insertions(+)
create mode 100755 tests/misc-tests/049-receive-write-encoded/test.sh
@@ -0,0 +1,114 @@+#!/bin/bash+#+# test that we can send and receive encoded writes for three modes of+# transparent compression: zlib, lzo, and zstd.++source"$TEST_TOP/common"++check_prereqmkfs.btrfs+check_prereqbtrfs++setup_root_helper+prepare_test_dev++here=`pwd`++# assumes the filesystem exists, and does mount, write, snapshot, send, unmount+# for the specified encoding option+send_one(){+localstr+localsubv+localsnap++algorithm="$1"+shift+str="$1"+shift++subv="subv-$algorithm"+snap="snap-$algorithm"++run_check_mount_test_dev"-o""compress-force=$algorithm"+cd"$TEST_MNT"||_fail"cannot chdir to TEST_MNT"++run_check$SUDO_HELPER"$TOP/btrfs"subvolumecreate"$subv"+run_check$SUDO_HELPERddif=/dev/zeroof="$subv/file1"bs=1Mcount=1+run_check$SUDO_HELPERddif=/dev/zeroof="$subv/file2"bs=500Kcount=1+run_check$SUDO_HELPER"$TOP/btrfs"subvolumesnapshot-r"$subv""$snap"+run_check$SUDO_HELPER"$TOP/btrfs"send-f"$str""$snap""$@"++cd"$here"||_fail"cannot chdir back to test directory"+run_check_umount_test_dev+}++receive_one(){+localstr+str="$1"+shift++run_check_mkfs_test_dev+run_check_mount_test_dev+run_check$SUDO_HELPER"$TOP/btrfs"receive"$@"-v-f"$str""$TEST_MNT"+run_check_umount_test_dev+run_checkrm-f--"$str"+}++test_one_write_encoded(){+localstr+localalgorithm+algorithm="$1"+shift+str="$here/stream-$algorithm.stream"++run_check_mkfs_test_dev+send_one"$algorithm""$str"--compressed-data+receive_one"$str""$@"+}++test_one_stream_v1(){+localstr+localalgorithm+algorithm="$1"+shift+str="$here/stream-$algorithm.stream"++run_check_mkfs_test_dev+send_one"$algorithm""$str"--stream-version1+receive_one"$str""$@"+}++test_mix_write_encoded(){+localstrzlib+localstrlzo+localstrzstd+strzlib="$here/stream-zlib.stream"+strlzo="$here/stream-lzo.stream"+strzstd="$here/stream-zstd.stream"++run_check_mkfs_test_dev++send_one"zlib""$strzlib"--compressed-data+send_one"lzo""$strlzo"--compressed-data+send_one"zstd""$strzstd"--compressed-data++receive_one"$strzlib"+receive_one"$strlzo"+receive_one"$strzstd"+}++test_one_write_encoded"zlib"+test_one_write_encoded"lzo"+test_one_write_encoded"zstd"++# with decompression forced+test_one_write_encoded"zlib""--force-decompress"+test_one_write_encoded"lzo""--force-decompress"+test_one_write_encoded"zstd""--force-decompress"++# send stream v1+test_one_stream_v1"zlib"+test_one_stream_v1"lzo"+test_one_stream_v1"zstd"++# files use a mix of compression algorithms+test_mix_write_encoded
From: Nikolay Borisov <hidden> Date: 2021-10-14 12:03:55
On 1.09.21 г. 20:00, Omar Sandoval wrote:
From: Omar Sandoval <redacted>
Encoded I/O in Btrfs needs to check a write with a given logical size
without an iov_iter that matches that size (because the iov_iter we have
is for the compressed data). So, factor out the parts of
generic_write_check() that don't need an iov_iter into a new
generic_write_checks_count() function and export that.
Signed-off-by: Omar Sandoval <redacted>
From: Nikolay Borisov <hidden> Date: 2021-10-14 12:05:57
On 1.09.21 г. 20:00, Omar Sandoval wrote:
From: Omar Sandoval <redacted>
btrfs_csum_one_bio() loops over each filesystem block in the bio while
keeping a cursor of its current logical position in the file in order to
look up the ordered extent to add the checksums to. However, this
doesn't make much sense for compressed extents, as a sector on disk does
not correspond to a sector of decompressed file data. It happens to work
because 1) the compressed bio always covers one ordered extent and 2)
the size of the bio is always less than the size of the ordered extent.
However, the second point will not always be true for encoded writes.
Let's add a boolean parameter to btrfs_csum_one_bio() to indicate that
it can assume that the bio only covers one ordered extent. Since we're
already changing the signature, let's get rid of the contig parameter
and make it implied by the offset parameter, similar to the change we
recently made to btrfs_lookup_bio_sums(). Additionally, let's rename
nr_sectors to blockcount to make it clear that it's the number of
filesystem blocks, not the number of 512-byte sectors.
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Omar Sandoval <redacted>
Code-wise this looks, though I don't know why we are guaranteed that a
compressed extent will only cover a single OE. But I trust you so:
Reviewed-by: Nikolay Borisov <redacted>
From: Nikolay Borisov <hidden> Date: 2021-10-14 12:54:48
On 1.09.21 г. 20:01, Omar Sandoval wrote:
From: Omar Sandoval <redacted>
Currently, an inline extent is always created after i_size is extended
from btrfs_dirty_pages(). However, for encoded writes, we only want to
update i_size after we successfully created the inline extent. Add an
update_i_size parameter to cow_file_range_inline() and
insert_inline_extent() and pass in the size of the extent rather than
determining it from i_size. Since the start parameter is always passed
as 0, get rid of it and simplify the logic in these two functions. While
we're here, let's document the requirements for creating an inline
extent.
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Omar Sandoval <redacted>
Overall I like where this is going, actually the diff is larger than it
should be because some refactoring is tucked in. If there would be yet
another submission with more substantial changes I'd like to see this
patch split into 2:
1. Patch does the refactoring
2. Adds the logic to update i_size alongside the addition of the
update_i_size parameter.
That will make the functional change a lot more obvious.
@@ -236,9 +236,10 @@ static int btrfs_init_inode_security(struct btrfs_trans_handle *trans,staticintinsert_inline_extent(structbtrfs_trans_handle*trans,structbtrfs_path*path,boolextent_inserted,structbtrfs_root*root,structinode*inode,-u64start,size_tsize,size_tcompressed_size,+size_tsize,size_tcompressed_size,intcompress_type,-structpage**compressed_pages)+structpage**compressed_pages,+boolupdate_i_size)
This function grows to 10 parameters, I think it's high time it took an
argument struct and each member can in turn be documented in this
structure, otherwise it's very unwieldy to work with.
@@ -327,7 +326,13 @@ static int insert_inline_extent(struct btrfs_trans_handle *trans, * before we unlock the pages. Otherwise we * could end up racing with unlink. */- BTRFS_I(inode)->disk_i_size = inode->i_size;+ i_size = i_size_read(inode);+ if (update_i_size && size > i_size) {+ i_size_write(inode, size);+ i_size = size;+ }+ BTRFS_I(inode)->disk_i_size = i_size;+ fail: return ret; }
@@ -338,35 +343,31 @@ static int insert_inline_extent(struct btrfs_trans_handle *trans, * does the checks required to make sure the data is small enough * to fit as an inline extent. */-static noinline int cow_file_range_inline(struct btrfs_inode *inode, u64 start,- u64 end, size_t compressed_size,+static noinline int cow_file_range_inline(struct btrfs_inode *inode, u64 size,+ size_t compressed_size, int compress_type,- struct page **compressed_pages)+ struct page **compressed_pages,+ bool update_i_size)
This function also has 6 parameters which is not on the extreme but I
have a feeling it also needs to be refactored to take an argument
structure.
{
struct btrfs_drop_extents_args drop_args = { 0 };
struct btrfs_root *root = inode->root;
struct btrfs_fs_info *fs_info = root->fs_info;
struct btrfs_trans_handle *trans;
- u64 isize = i_size_read(&inode->vfs_inode);
- u64 actual_end = min(end + 1, isize);
- u64 inline_len = actual_end - start;
- u64 aligned_end = ALIGN(end, fs_info->sectorsize);
- u64 data_len = inline_len;
+ u64 data_len = compressed_size ? compressed_size : size;
int ret;
struct btrfs_path *path;
- if (compressed_size)
- data_len = compressed_size;
-
- if (start > 0 ||
- actual_end > fs_info->sectorsize ||
+ /*
+ * We can create an inline extent if it ends at or beyond the current
+ * i_size, is no larger than a sector (decompressed), and the (possibly
+ * compressed) data fits in a leaf and the configured maximum inline
+ * size.
+ */
+ if (size < i_size_read(&inode->vfs_inode) ||
+ size > fs_info->sectorsize ||
data_len > BTRFS_MAX_INLINE_DATA_SIZE(fs_info) ||
- (!compressed_size &&
- (actual_end & (fs_info->sectorsize - 1)) == 0) ||
- end + 1 < isize ||
- data_len > fs_info->max_inline) {
+ data_len > fs_info->max_inline)
return 1;
With this change you allow to write more than PAGE_SIZE data in case of
a compressed extents, Qu said he had some concerns regarding this. I'm
CCing him so this can be discussed on this submission.
From: Nikolay Borisov <hidden> Date: 2021-10-15 09:42:42
On 1.09.21 г. 20:01, Omar Sandoval wrote:
From: Omar Sandoval <redacted>
In order to allow sending and receiving compressed data without
decompressing it, we need an interface to write pre-compressed data
directly to the filesystem and the matching interface to read compressed
data without decompressing it. This adds the definitions for ioctls to
do that and detailed explanations of how to use them.
Signed-off-by: Omar Sandoval <redacted>
One minor nit below but otherwise LGTM:
Reviewed-by: Nikolay Borisov <redacted>
<snip>
+struct btrfs_ioctl_encoded_io_args {
+ /* Input parameters for both reads and writes. */
+
+ /*
+ * iovecs containing encoded data.
+ *
+ * For reads, if the size of the encoded data is larger than the sum of
+ * iov[n].iov_len for 0 <= n < iovcnt, then the ioctl fails with
+ * ENOBUFS.
+ *
+ * For writes, the size of the encoded data is the sum of iov[n].iov_len
+ * for 0 <= n < iovcnt. This must be less than 128 KiB (this limit may
+ * increase in the future). This must also be less than or equal to
+ * unencoded_len.
+ */
+ const struct iovec __user *iov;
+ /* Number of iovecs. */
+ unsigned long iovcnt;
+ /*
+ * Offset in file.
+ *
+ * For writes, must be aligned to the sector size of the filesystem.
+ */
+ __s64 offset;
+ /* Currently must be zero. */
+ __u64 flags;
+
nit: A comment stating that the output params begin here could be added.
From: Nikolay Borisov <hidden> Date: 2021-10-15 11:45:50
On 1.09.21 г. 20:01, Omar Sandoval wrote:
quoted hunk
From: Omar Sandoval <redacted>
There are 4 main cases:
1. Inline extents: we copy the data straight out of the extent buffer.
2. Hole/preallocated extents: we fill in zeroes.
3. Regular, uncompressed extents: we read the sectors we need directly
from disk.
4. Regular, compressed extents: we read the entire compressed extent
from disk and indicate what subset of the decompressed extent is in
the file.
This initial implementation simplifies a few things that can be improved
in the future:
- We hold the inode lock during the operation.
- Cases 1, 3, and 4 allocate temporary memory to read into before
copying out to userspace.
- We don't do read repair, because it turns out that read repair is
currently broken for compressed data.
Signed-off-by: Omar Sandoval <redacted>
---
fs/btrfs/ctree.h | 4 +
fs/btrfs/inode.c | 489 +++++++++++++++++++++++++++++++++++++++++++++++
fs/btrfs/ioctl.c | 111 +++++++++++
3 files changed, 604 insertions(+)
nit: The gist of this function is to check the csum so how about
renaming it to btrfs_encoded_read_verify_csum
<snip>
+
+static void btrfs_encoded_read_endio(struct bio *bio)
+{
+ struct btrfs_encoded_read_private *priv = bio->bi_private;
+ struct btrfs_io_bio *io_bio = btrfs_io_bio(bio);
+ blk_status_t status;
+
+ status = btrfs_encoded_read_check_bio(io_bio);
+ if (status) {
+ /*
+ * The memory barrier implied by the atomic_dec_return() here
+ * pairs with the memory barrier implied by the
+ * atomic_dec_return() or io_wait_event() in
nit: I think atomic_dec_return in read_regular_fill_pages is
inconsequential, what we want to ensure is that when the caller of
io_wait_event is woken up by this thread it will observe the
priv->status, which it will, because the atomic-dec_return in this
function has paired with the general barrier interpolated by wait_event.
So for brevity just leave the text to say "by io_wait_event".
+ * btrfs_encoded_read_regular_fill_pages() to ensure that this
+ * write is observed before the load of status in
+ * btrfs_encoded_read_regular_fill_pages().
+ */
+ WRITE_ONCE(priv->status, status);
+ }
+ if (!atomic_dec_return(&priv->pending))
+ wake_up(&priv->wait);
+ btrfs_io_bio_free_csum(io_bio);
+ bio_put(bio);
+}
From: Nikolay Borisov <hidden> Date: 2021-10-18 11:10:07
On 1.09.21 г. 20:01, Omar Sandoval wrote:
From: Omar Sandoval <redacted>
For encoded writes, we need the raw pages for reading compressed data
directly via a bio. So, replace kvmalloc() with vmap() so we have access
to the raw pages. 144k is large enough that it usually gets allocated
with vmalloc(), anyways.
Signed-off-by: Omar Sandoval <redacted>
From: Nikolay Borisov <hidden> Date: 2021-10-18 11:59:12
On 1.09.21 г. 20:01, Omar Sandoval wrote:
From: Omar Sandoval <redacted>
Now that all of the pieces are in place, we can use the ENCODED_WRITE
command to send compressed extents when appropriate.
Signed-off-by: Omar Sandoval <redacted>
Overall looks sane but consider some of the nits below.
<snip>
nit: The whole data_offset warrants a comment here, since send_buf is
now mapped from send_buf_pages, so all the TLV you've put before are
actually stored in the beginning of send_buf_pages, so by doing the
above you ensure the data write begins on a clean page boundary ...
nit: Instead of sending around a btrfs_path struct around and
"polluting" callees to deal with the oddities of our btree interface i.e
btrfs_item_ptr et al. Why not refactor the code so that when we know we
are about to send an extent data simply initialize some struct
extent_info with all the necessary data items: extent type, compression
type, based on the extent type properly initialize a size attribute etc
and pass that. Right now you have send_extent_data fiddling with
path->nodes[0], then based on that you either call
send_encoded_inline_extent or send_encoded_extent, instead pass
extent_info to send_extent_data/clone_range and be done with it.
<snip>
From: Nikolay Borisov <hidden> Date: 2021-10-18 12:44:26
On 1.09.21 г. 20:01, Omar Sandoval wrote:
From: Omar Sandoval <redacted>
Now that the new support is implemented, allow the ioctl to accept the
flags and update the version in sysfs.
This seems like an appropriate place to bring up the discussion about
versioned streams. SO instead of adding a BTRFS_SEND_FLAG_STREAM_V2
flag, which implies that for the next version we have to add
BTRFS_SEND_FLAG_STREAM_V3 etc. Why not generalize the flag to
BTRFS_SEND_FLAG_STREAM_VERSIONED and carve an u32 from one of the
reserved fields so that in the future we simply increment the version field?
From: Nikolay Borisov <hidden> Date: 2021-10-18 12:46:23
On 1.09.21 г. 20:01, Omar Sandoval wrote:
From: Omar Sandoval <redacted>
This adds the definitions of the new commands for send stream version 2
and their respective attributes: fallocate, FS_IOC_SETFLAGS (a.k.a.
chattr), and encoded writes. It also documents two changes to the send
stream format in v2: the receiver shouldn't assume a maximum command
size, and the DATA attribute is encoded differently to allow for writes
larger than 64k. These will be implemented in subsequent changes, and
then the ioctl will accept the new flags.
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Omar Sandoval <redacted>
From: Nikolay Borisov <hidden> Date: 2021-10-18 12:57:22
On 1.09.21 г. 20:01, Omar Sandoval wrote:
From: Omar Sandoval <redacted>
The length field of the send stream TLV header is 16 bits. This means
that the maximum amount of data that can be sent for one write is 64k
minus one. However, encoded writes must be able to send the maximum
compressed extent (128k) in one command. To support this, send stream
version 2 encodes the DATA attribute differently: it has no length
field, and the length is implicitly up to the end of containing command
(which has a 32-bit length field). Although this is necessary for
encoded writes, normal writes can benefit from it, too.
For v2, let's bump up the send buffer to the maximum compressed extent
size plus 16k for the other metadata (144k total). Since this will most
likely be vmalloc'd (and always will be after the next commit), we round
it up to the next page since we might as well use the rest of the page
on systems with >16k pages.
Signed-off-by: Omar Sandoval <redacted>
From: Nikolay Borisov <hidden> Date: 2021-10-18 15:12:19
On 1.09.21 г. 20:01, Omar Sandoval wrote:
quoted hunk
From: Omar Sandoval <redacted>
This adds the definitions of the new commands for send stream version 2
and their respective attributes: fallocate, FS_IOC_SETFLAGS (a.k.a.
chattr), and encoded writes. It also documents two changes to the send
stream format in v2: the receiver shouldn't assume a maximum command
size, and the DATA attribute is encoded differently to allow for writes
larger than 64k. These will be implemented in subsequent changes, and
then the ioctl will accept the new flags.
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Omar Sandoval <redacted>
---
fs/btrfs/send.c | 2 +-
fs/btrfs/send.h | 30 +++++++++++++++++++++++++++++-
include/uapi/linux/btrfs.h | 13 +++++++++++++
3 files changed, 43 insertions(+), 2 deletions(-)
Now that I think more about this, it would be best if this logic is
actually codified in the code. I.e first set of SEND_A_DATA would set
some bool/flag in the sctx and subsequent calls would be able to
ASSERT/WARN ?
On Thu, Oct 14, 2021 at 03:05:54PM +0300, Nikolay Borisov wrote:
On 1.09.21 г. 20:00, Omar Sandoval wrote:
quoted
From: Omar Sandoval <redacted>
btrfs_csum_one_bio() loops over each filesystem block in the bio while
keeping a cursor of its current logical position in the file in order to
look up the ordered extent to add the checksums to. However, this
doesn't make much sense for compressed extents, as a sector on disk does
not correspond to a sector of decompressed file data. It happens to work
because 1) the compressed bio always covers one ordered extent and 2)
the size of the bio is always less than the size of the ordered extent.
However, the second point will not always be true for encoded writes.
Let's add a boolean parameter to btrfs_csum_one_bio() to indicate that
it can assume that the bio only covers one ordered extent. Since we're
already changing the signature, let's get rid of the contig parameter
and make it implied by the offset parameter, similar to the change we
recently made to btrfs_lookup_bio_sums(). Additionally, let's rename
nr_sectors to blockcount to make it clear that it's the number of
filesystem blocks, not the number of 512-byte sectors.
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Omar Sandoval <redacted>
Code-wise this looks, though I don't know why we are guaranteed that a
compressed extent will only cover a single OE.
See submit_compressed_extents(): we always add an ordered extent that
covers the whole range that we're submitting, and then
btrfs_submit_compressed_write() creates one or more bios that are
subsets of that range.
But I trust you so:
Reviewed-by: Nikolay Borisov <redacted>
On Fri, Oct 15, 2021 at 12:42:37PM +0300, Nikolay Borisov wrote:
On 1.09.21 г. 20:01, Omar Sandoval wrote:
quoted
From: Omar Sandoval <redacted>
In order to allow sending and receiving compressed data without
decompressing it, we need an interface to write pre-compressed data
directly to the filesystem and the matching interface to read compressed
data without decompressing it. This adds the definitions for ioctls to
do that and detailed explanations of how to use them.
Signed-off-by: Omar Sandoval <redacted>
One minor nit below but otherwise LGTM:
Reviewed-by: Nikolay Borisov <redacted>
<snip>
quoted
+struct btrfs_ioctl_encoded_io_args {
+ /* Input parameters for both reads and writes. */
+
+ /*
+ * iovecs containing encoded data.
+ *
+ * For reads, if the size of the encoded data is larger than the sum of
+ * iov[n].iov_len for 0 <= n < iovcnt, then the ioctl fails with
+ * ENOBUFS.
+ *
+ * For writes, the size of the encoded data is the sum of iov[n].iov_len
+ * for 0 <= n < iovcnt. This must be less than 128 KiB (this limit may
+ * increase in the future). This must also be less than or equal to
+ * unencoded_len.
+ */
+ const struct iovec __user *iov;
+ /* Number of iovecs. */
+ unsigned long iovcnt;
+ /*
+ * Offset in file.
+ *
+ * For writes, must be aligned to the sector size of the filesystem.
+ */
+ __s64 offset;
+ /* Currently must be zero. */
+ __u64 flags;
+
nit: A comment stating that the output params begin here could be added.
There is a comment in this spot:
/*
* For reads, the following members are filled in with the metadata for
* the encoded data.
* For writes, the following members must be set to the metadata for the
* encoded data.
*/
I'll clarify it to:
/*
* For reads, the following members are output parameters that will
* contain the returned metadata for the encoded data.
* For writes, the following members must be set to the metadata for the
* encoded data.
*/
On Mon, Oct 18, 2021 at 03:44:23PM +0300, Nikolay Borisov wrote:
On 1.09.21 г. 20:01, Omar Sandoval wrote:
quoted
From: Omar Sandoval <redacted>
Now that the new support is implemented, allow the ioctl to accept the
flags and update the version in sysfs.
This seems like an appropriate place to bring up the discussion about
versioned streams. SO instead of adding a BTRFS_SEND_FLAG_STREAM_V2
flag, which implies that for the next version we have to add
BTRFS_SEND_FLAG_STREAM_V3 etc. Why not generalize the flag to
BTRFS_SEND_FLAG_STREAM_VERSIONED and carve an u32 from one of the
reserved fields so that in the future we simply increment the version field?
On Mon, Oct 18, 2021 at 06:11:54PM +0300, Nikolay Borisov wrote:
On 1.09.21 г. 20:01, Omar Sandoval wrote:
quoted
From: Omar Sandoval <redacted>
This adds the definitions of the new commands for send stream version 2
and their respective attributes: fallocate, FS_IOC_SETFLAGS (a.k.a.
chattr), and encoded writes. It also documents two changes to the send
stream format in v2: the receiver shouldn't assume a maximum command
size, and the DATA attribute is encoded differently to allow for writes
larger than 64k. These will be implemented in subsequent changes, and
then the ioctl will accept the new flags.
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Omar Sandoval <redacted>
---
fs/btrfs/send.c | 2 +-
fs/btrfs/send.h | 30 +++++++++++++++++++++++++++++-
include/uapi/linux/btrfs.h | 13 +++++++++++++
3 files changed, 43 insertions(+), 2 deletions(-)
Now that I think more about this, it would be best if this logic is
actually codified in the code. I.e first set of SEND_A_DATA would set
some bool/flag in the sctx and subsequent calls would be able to
ASSERT/WARN ?
I suppose I could do something like this, is that what you had in mind?
On Fri, Oct 15, 2021 at 02:45:46PM +0300, Nikolay Borisov wrote:
On 1.09.21 г. 20:01, Omar Sandoval wrote:
quoted
From: Omar Sandoval <redacted>
There are 4 main cases:
1. Inline extents: we copy the data straight out of the extent buffer.
2. Hole/preallocated extents: we fill in zeroes.
3. Regular, uncompressed extents: we read the sectors we need directly
from disk.
4. Regular, compressed extents: we read the entire compressed extent
from disk and indicate what subset of the decompressed extent is in
the file.
This initial implementation simplifies a few things that can be improved
in the future:
- We hold the inode lock during the operation.
- Cases 1, 3, and 4 allocate temporary memory to read into before
copying out to userspace.
- We don't do read repair, because it turns out that read repair is
currently broken for compressed data.
Signed-off-by: Omar Sandoval <redacted>
---
fs/btrfs/ctree.h | 4 +
fs/btrfs/inode.c | 489 +++++++++++++++++++++++++++++++++++++++++++++++
fs/btrfs/ioctl.c | 111 +++++++++++
3 files changed, 604 insertions(+)
nit: The gist of this function is to check the csum so how about
renaming it to btrfs_encoded_read_verify_csum
Good point, will do.
quoted
+
+static void btrfs_encoded_read_endio(struct bio *bio)
+{
+ struct btrfs_encoded_read_private *priv = bio->bi_private;
+ struct btrfs_io_bio *io_bio = btrfs_io_bio(bio);
+ blk_status_t status;
+
+ status = btrfs_encoded_read_check_bio(io_bio);
+ if (status) {
+ /*
+ * The memory barrier implied by the atomic_dec_return() here
+ * pairs with the memory barrier implied by the
+ * atomic_dec_return() or io_wait_event() in
nit: I think atomic_dec_return in read_regular_fill_pages is
inconsequential, what we want to ensure is that when the caller of
io_wait_event is woken up by this thread it will observe the
priv->status, which it will, because the atomic-dec_return in this
function has paired with the general barrier interpolated by wait_event.
So for brevity just leave the text to say "by io_wait_event".
Considering that there is a code path in
btrfs_encoded_read_regular_fill_pages() where atomic_dec_return() is
called but io_wait_event() isn't, I think it's important to mention
both. (This path should be fairly rare, since it can only happen if all
of the bios are completed before the submitting thread checks the
pending counter. But, it's a possible code path, and I wouldn't want
someone reading the comment to be confused and think that we're missing
a barrier in that case.)
quoted
+ * btrfs_encoded_read_regular_fill_pages() to ensure that this
+ * write is observed before the load of status in
+ * btrfs_encoded_read_regular_fill_pages().
+ */
+ WRITE_ONCE(priv->status, status);
+ }
+ if (!atomic_dec_return(&priv->pending))
+ wake_up(&priv->wait);
+ btrfs_io_bio_free_csum(io_bio);
+ bio_put(bio);
+}
On Mon, Oct 18, 2021 at 02:59:08PM +0300, Nikolay Borisov wrote:
On 1.09.21 г. 20:01, Omar Sandoval wrote:
quoted
From: Omar Sandoval <redacted>
Now that all of the pieces are in place, we can use the ENCODED_WRITE
command to send compressed extents when appropriate.
Signed-off-by: Omar Sandoval <redacted>
Overall looks sane but consider some of the nits below.
<snip>
nit: The whole data_offset warrants a comment here, since send_buf is
now mapped from send_buf_pages, so all the TLV you've put before are
actually stored in the beginning of send_buf_pages, so by doing the
above you ensure the data write begins on a clean page boundary ...
nit: Instead of sending around a btrfs_path struct around and
"polluting" callees to deal with the oddities of our btree interface i.e
btrfs_item_ptr et al. Why not refactor the code so that when we know we
are about to send an extent data simply initialize some struct
extent_info with all the necessary data items: extent type, compression
type, based on the extent type properly initialize a size attribute etc
and pass that. Right now you have send_extent_data fiddling with
path->nodes[0], then based on that you either call
send_encoded_inline_extent or send_encoded_extent, instead pass
extent_info to send_extent_data/clone_range and be done with it.
I don't like this for a few reasons:
* An extra "struct extent_info" layer of abstraction would just be extra
cognitive overhead. I hate having to trace back where the fields in
some struct came from when it's information that's readily available
in more well-known data structures.
* send_encoded_inline_extent() (called by send_extent_data()) needs the
btrfs_path in order to get the inline data anyways.
* clone_range() also already deals with btrfs_paths, so it's not new.
From: Nikolay Borisov <hidden> Date: 2021-10-19 07:02:05
On 18.10.21 г. 21:58, Omar Sandoval wrote:
I suppose I could do something like this, is that what you had in mind?
Yes, however I think it needs to be augmented a bit like making this
member populated/checked only if V2 is being used? But that's generally
what I had in mind.
From: Nikolay Borisov <hidden> Date: 2021-10-20 13:49:42
On 1.09.21 г. 20:01, Omar Sandoval wrote:
quoted hunk
From: Boris Burkov <redacted>
An encoded extent can be up to 128K in length, which exceeds the largest
value expressible by the current send stream format's 16 bit tlv_len
field. Since encoded writes cannot be split into multiple writes by
btrfs send, the send stream format must change to accommodate encoded
writes.
Supporting this changed format requires retooling how we store the
commands we have processed. Since we can no longer use btrfs_tlv_header
to describe every attribute, we define a new struct btrfs_send_attribute
which has a 32 bit length field, and use that to store the attribute
information needed for receive processing. This is transparent to users
of the various TLV_GET macros.
Signed-off-by: Boris Burkov <redacted>
---
common/send-stream.c | 34 +++++++++++++++++++++++++---------
1 file changed, 25 insertions(+), 9 deletions(-)
This is subtle and it took me a couple of minutes to get it at first.
Currently cmds_attrs holds an array of pointers into the command buffer,
with every pointer being the beginning of the tlv_header, whilst with
your change cmd_attr now holds actual btrfs_send_attribute structures
(52 bytes vs sizeof(uintptr_t) bytes before). So this increases the
overall size of btrfs_send_stream because with your version of the code
you parse the type/length fields and store them directly in the send
attribute structure at command parse time rather than just referring to
the raw command buffer during read_cmd and referring to them during
attribute parsing.
This might seem superficial but this kind of change should really be
mentioned explicitly in the changelog to better prepare reviewers what
to expect.
OTOH the code LGTM and actually now it seems less tricky than before so:
Reviewed-by: Nikolay Borisov <redacted>
David if you deem it necessary adjust the commit message appropriately.
From: Nikolay Borisov <hidden> Date: 2021-10-20 14:09:23
On 1.09.21 г. 20:01, Omar Sandoval wrote:
From: Boris Burkov <redacted>
In send stream v2, write commands can now be an arbitrary size. For that
reason, we can no longer allocate a fixed array in sctx for read_cmd.
Instead, read_cmd dynamically allocates sctx->read_buf. To avoid
needless reallocations, we reuse read_buf between read_cmd calls by also
keeping track of the size of the allocated buffer in sctx->read_buf_sz.
We do the first allocation of the old default size at the start of
processing the stream, and we only reallocate if we encounter a command
that needs a larger buffer.
Signed-off-by: Boris Burkov <redacted>
---
common/send-stream.c | 55 ++++++++++++++++++++++++++++----------------
send.h | 2 +-
2 files changed, 36 insertions(+), 21 deletions(-)
<snip>
quoted hunk
@@ -124,18 +125,22 @@ static int read_cmd(struct btrfs_send_stream *sctx) goto out; }- sctx->cmd_hdr = (struct btrfs_cmd_header *)sctx->read_buf;- cmd = le16_to_cpu(sctx->cmd_hdr->cmd);- cmd_len = le32_to_cpu(sctx->cmd_hdr->len);-- if (cmd_len + sizeof(*sctx->cmd_hdr) >= sizeof(sctx->read_buf)) {- ret = -EINVAL;- error("command length %u too big for buffer %zu",- cmd_len, sizeof(sctx->read_buf));- goto out;+ cmd_hdr = (struct btrfs_cmd_header *)sctx->read_buf;+ cmd_len = le32_to_cpu(cmd_hdr->len);+ cmd = le16_to_cpu(cmd_hdr->cmd);+ buf_len = sizeof(*cmd_hdr) + cmd_len;+ if (sctx->read_buf_sz < buf_len) {+ sctx->read_buf = realloc(sctx->read_buf, buf_len);+ if (!sctx->read_buf) {
nit: This is prone to a memory leak, because according to
https://en.cppreference.com/w/c/memory/realloc
If there is not enough memory, the old memory block is not freed and
null pointer is returned.
This means if realloc fails it will overwrite sctx->read_buf with NULL,
yet the old memory won't be freed which will cause a memory leak. It can
be argued that's not critical since we'll very quickly terminate the
program afterwards but still.
<snip>
From: Nikolay Borisov <hidden> Date: 2021-10-20 14:35:46
On 20.10.21 г. 17:09, Nikolay Borisov wrote:
On 1.09.21 г. 20:01, Omar Sandoval wrote:
quoted
From: Boris Burkov <redacted>
In send stream v2, write commands can now be an arbitrary size. For that
nit: Actually can't commands really be up-to BTRFS_MAX_COMPRESSED + 16K
really or are we going to leave this as an implementation detail? I'm
fine either way but looking at the changelog of patch 12 in the kernel
series doesn't really mention of arbitrary size, instead it explicitly
is talking about sending the max compressed extent size (128K) + some
space for metadata (the 16K above).
quoted
reason, we can no longer allocate a fixed array in sctx for read_cmd.
Instead, read_cmd dynamically allocates sctx->read_buf. To avoid
needless reallocations, we reuse read_buf between read_cmd calls by also
keeping track of the size of the allocated buffer in sctx->read_buf_sz.
We do the first allocation of the old default size at the start of
processing the stream, and we only reallocate if we encounter a command
that needs a larger buffer.
Signed-off-by: Boris Burkov <redacted>
---
common/send-stream.c | 55 ++++++++++++++++++++++++++++----------------
send.h | 2 +-
2 files changed, 36 insertions(+), 21 deletions(-)
<snip>
quoted
@@ -124,18 +125,22 @@ static int read_cmd(struct btrfs_send_stream *sctx) goto out; }- sctx->cmd_hdr = (struct btrfs_cmd_header *)sctx->read_buf;- cmd = le16_to_cpu(sctx->cmd_hdr->cmd);- cmd_len = le32_to_cpu(sctx->cmd_hdr->len);-- if (cmd_len + sizeof(*sctx->cmd_hdr) >= sizeof(sctx->read_buf)) {- ret = -EINVAL;- error("command length %u too big for buffer %zu",- cmd_len, sizeof(sctx->read_buf));- goto out;+ cmd_hdr = (struct btrfs_cmd_header *)sctx->read_buf;+ cmd_len = le32_to_cpu(cmd_hdr->len);+ cmd = le16_to_cpu(cmd_hdr->cmd);+ buf_len = sizeof(*cmd_hdr) + cmd_len;+ if (sctx->read_buf_sz < buf_len) {+ sctx->read_buf = realloc(sctx->read_buf, buf_len);+ if (!sctx->read_buf) {
nit: This is prone to a memory leak, because according to
https://en.cppreference.com/w/c/memory/realloc
If there is not enough memory, the old memory block is not freed and
null pointer is returned.
This means if realloc fails it will overwrite sctx->read_buf with NULL,
yet the old memory won't be freed which will cause a memory leak. It can
be argued that's not critical since we'll very quickly terminate the
program afterwards but still.
<snip>
From: Nikolay Borisov <hidden> Date: 2021-10-20 14:36:06
On 1.09.21 г. 20:01, Omar Sandoval wrote:
From: Boris Burkov <redacted>
The new format privileges the BTRFS_SEND_A_DATA attribute by
guaranteeing it will always be the last attribute in any command that
needs it, and by implicitly encoding the data length as the difference
between the total command length in the command header and the sizes of
the rest of the attributes (and of course the tlv_type identifying the
DATA attribute). To parse the new stream, we must read the tlv_type and
if it is not DATA, we proceed normally, but if it is DATA, we don't
parse a tlv_len but simply compute the length.
In addition, we add some bounds checking when parsing each chunk of
data, as well as for the tlv_len itself.
Signed-off-by: Boris Burkov <redacted>
From: Nikolay Borisov <hidden> Date: 2021-10-20 14:38:09
On 1.09.21 г. 20:01, Omar Sandoval wrote:
From: Boris Burkov <redacted>
Send stream v2 adds three commands and several attributes associated to
those commands. Before we implement processing them, add all the
commands and attributes. This avoids leaving the enums in an
intermediate state that doesn't correspond to any version of send
stream.
Signed-off-by: Boris Burkov <redacted>
On Wed, Oct 20, 2021 at 05:35:42PM +0300, Nikolay Borisov wrote:
On 20.10.21 г. 17:09, Nikolay Borisov wrote:
quoted
On 1.09.21 г. 20:01, Omar Sandoval wrote:
quoted
From: Boris Burkov <redacted>
In send stream v2, write commands can now be an arbitrary size. For that
nit: Actually can't commands really be up-to BTRFS_MAX_COMPRESSED + 16K
really or are we going to leave this as an implementation detail? I'm
fine either way but looking at the changelog of patch 12 in the kernel
series doesn't really mention of arbitrary size, instead it explicitly
is talking about sending the max compressed extent size (128K) + some
space for metadata (the 16K above).
Patch 10 mentions it in the changelog: "It also documents two changes to the
send stream format in v2: the receiver shouldn't assume a maximum command size,
and the DATA attribute is encoded differently to allow for writes larger than
64k".
And in send.h:
-#define BTRFS_SEND_BUF_SIZE SZ_64K
+/*
+ * In send stream v1, no command is larger than 64k. In send stream v2, no limit
+ * should be assumed.
+ */
+#define BTRFS_SEND_BUF_SIZE_V1 SZ_64K
You're correct that right now the limit is BTRFS_MAX_COMPRESSED + 16k,
but I think it's better if userspace doesn't make any assumptions about
that in case we want to send larger commands in the future.
quoted
quoted
reason, we can no longer allocate a fixed array in sctx for read_cmd.
Instead, read_cmd dynamically allocates sctx->read_buf. To avoid
needless reallocations, we reuse read_buf between read_cmd calls by also
keeping track of the size of the allocated buffer in sctx->read_buf_sz.
We do the first allocation of the old default size at the start of
processing the stream, and we only reallocate if we encounter a command
that needs a larger buffer.
Signed-off-by: Boris Burkov <redacted>
---
common/send-stream.c | 55 ++++++++++++++++++++++++++++----------------
send.h | 2 +-
2 files changed, 36 insertions(+), 21 deletions(-)
<snip>
quoted
@@ -124,18 +125,22 @@ static int read_cmd(struct btrfs_send_stream *sctx) goto out; }- sctx->cmd_hdr = (struct btrfs_cmd_header *)sctx->read_buf;- cmd = le16_to_cpu(sctx->cmd_hdr->cmd);- cmd_len = le32_to_cpu(sctx->cmd_hdr->len);-- if (cmd_len + sizeof(*sctx->cmd_hdr) >= sizeof(sctx->read_buf)) {- ret = -EINVAL;- error("command length %u too big for buffer %zu",- cmd_len, sizeof(sctx->read_buf));- goto out;+ cmd_hdr = (struct btrfs_cmd_header *)sctx->read_buf;+ cmd_len = le32_to_cpu(cmd_hdr->len);+ cmd = le16_to_cpu(cmd_hdr->cmd);+ buf_len = sizeof(*cmd_hdr) + cmd_len;+ if (sctx->read_buf_sz < buf_len) {+ sctx->read_buf = realloc(sctx->read_buf, buf_len);+ if (!sctx->read_buf) {
nit: This is prone to a memory leak, because according to
https://en.cppreference.com/w/c/memory/realloc
If there is not enough memory, the old memory block is not freed and
null pointer is returned.
This means if realloc fails it will overwrite sctx->read_buf with NULL,
yet the old memory won't be freed which will cause a memory leak. It can
be argued that's not critical since we'll very quickly terminate the
program afterwards but still.
On Wed, Oct 20, 2021 at 04:49:38PM +0300, Nikolay Borisov wrote:
On 1.09.21 г. 20:01, Omar Sandoval wrote:
quoted
From: Boris Burkov <redacted>
An encoded extent can be up to 128K in length, which exceeds the largest
value expressible by the current send stream format's 16 bit tlv_len
field. Since encoded writes cannot be split into multiple writes by
btrfs send, the send stream format must change to accommodate encoded
writes.
Supporting this changed format requires retooling how we store the
commands we have processed. Since we can no longer use btrfs_tlv_header
to describe every attribute, we define a new struct btrfs_send_attribute
which has a 32 bit length field, and use that to store the attribute
information needed for receive processing. This is transparent to users
of the various TLV_GET macros.
Signed-off-by: Boris Burkov <redacted>
---
common/send-stream.c | 34 +++++++++++++++++++++++++---------
1 file changed, 25 insertions(+), 9 deletions(-)
This is subtle and it took me a couple of minutes to get it at first.
Currently cmds_attrs holds an array of pointers into the command buffer,
with every pointer being the beginning of the tlv_header, whilst with
your change cmd_attr now holds actual btrfs_send_attribute structures
(52 bytes vs sizeof(uintptr_t) bytes before). So this increases the
overall size of btrfs_send_stream because with your version of the code
you parse the type/length fields and store them directly in the send
attribute structure at command parse time rather than just referring to
the raw command buffer during read_cmd and referring to them during
attribute parsing.
This might seem superficial but this kind of change should really be
mentioned explicitly in the changelog to better prepare reviewers what
to expect.
OTOH the code LGTM and actually now it seems less tricky than before so:
Reviewed-by: Nikolay Borisov <redacted>
David if you deem it necessary adjust the commit message appropriately.
I clarified the second paragraph to:
Supporting this changed format requires retooling how we store the
commands we have processed. We currently store pointers to the struct
btrfs_tlv_headers in the command buffer. This is not sufficient to
represent the new BTRFS_SEND_A_DATA format. Instead, parse the attribute
headers and store them in a new struct btrfs_send_attribute which has a
32-bit length field. This is transparent to users of the various TLV_GET
macros.
From: Nikolay Borisov <hidden> Date: 2021-10-21 12:44:57
On 1.09.21 г. 20:00, Omar Sandoval wrote:
From: Omar Sandoval <redacted>
Currently, we only create ordered extents when ram_bytes == num_bytes
and offset == 0. However, RWF_ENCODED writes may create extents which
Change RWF_ENCODED to simply encoded as we no longer rely on RWF flags,
same thing for the changelog in the next patch.
<snip>
From: Nikolay Borisov <hidden> Date: 2021-10-21 13:33:05
On 1.09.21 г. 20:01, Omar Sandoval wrote:
From: Boris Burkov <redacted>
<snip>
+/* Data is not compressed. */
+#define BTRFS_ENCODED_IO_COMPRESSION_NONE 0
+/* Data is compressed as a single zlib stream. */
+#define BTRFS_ENCODED_IO_COMPRESSION_ZLIB 1
+/*
+ * Data is compressed as a single zstd frame with the windowLog compression
+ * parameter set to no more than 17.
+ */
+#define BTRFS_ENCODED_IO_COMPRESSION_ZSTD 2
+/*
+ * Data is compressed page by page (using the page size indicated by the name of
+ * the constant) with LZO1X and wrapped in the format documented in
+ * fs/btrfs/lzo.c. For writes, the compression page size must match the
+ * filesystem page size.
+ */
+#define BTRFS_ENCODED_IO_COMPRESSION_LZO_4K 3
+#define BTRFS_ENCODED_IO_COMPRESSION_LZO_8K 4
+#define BTRFS_ENCODED_IO_COMPRESSION_LZO_16K 5
+#define BTRFS_ENCODED_IO_COMPRESSION_LZO_32K 6
+#define BTRFS_ENCODED_IO_COMPRESSION_LZO_64K 7
+#define BTRFS_ENCODED_IO_COMPRESSION_TYPES 8
nit: Make those an enum ? Same applies for the kernel counterpart patch.
quoted hunk
+
+/* Data is not encrypted. */
+#define BTRFS_ENCODED_IO_ENCRYPTION_NONE 0
+#define BTRFS_ENCODED_IO_ENCRYPTION_TYPES 1
+
/* Error codes as returned by the kernel */
enum btrfs_err_code {
notused,
From: Nikolay Borisov <hidden> Date: 2021-10-21 14:21:08
On 1.09.21 г. 20:01, Omar Sandoval wrote:
From: Boris Burkov <redacted>
Send stream v2 can emit fallocate commands, so receive must support them
as well. The implementation simply passes along the arguments to the
syscall. Note that mode is encoded as a u32 in send stream but fallocate
takes an int, so there is a unsigned->signed conversion there.
Signed-off-by: Boris Burkov <redacted>
Reviewed-by: Nikolay Borisov <redacted>
However, kernel support for this hasn't landed, the kernel counterpart
patches add definitions but don't actually implement the code. BY the
looks of it it would seem that the proper send stream versioning could
be added first before any of this code lands. In this case we can simply
have the encoded writes stuff as protocol version 2 and leave the rest
of the commands for v3 for example.
From: Nikolay Borisov <hidden> Date: 2021-10-21 14:22:29
On 1.09.21 г. 20:01, Omar Sandoval wrote:
From: Boris Burkov <redacted>
In send stream v2, send can emit a command for setting inode flags via
the setflags ioctl. Pass the flags attribute through to the ioctl call
in receive.
Signed-off-by: Boris Burkov <redacted>
Revewed-by: Nikolay Borisov [off-list ref]
Same remark about the missing kernel implementation.
On Thu, Oct 21, 2021 at 04:33:00PM +0300, Nikolay Borisov wrote:
On 1.09.21 г. 20:01, Omar Sandoval wrote:
quoted
From: Boris Burkov <redacted>
<snip>
quoted
+/* Data is not compressed. */
+#define BTRFS_ENCODED_IO_COMPRESSION_NONE 0
+/* Data is compressed as a single zlib stream. */
+#define BTRFS_ENCODED_IO_COMPRESSION_ZLIB 1
+/*
+ * Data is compressed as a single zstd frame with the windowLog compression
+ * parameter set to no more than 17.
+ */
+#define BTRFS_ENCODED_IO_COMPRESSION_ZSTD 2
+/*
+ * Data is compressed page by page (using the page size indicated by the name of
+ * the constant) with LZO1X and wrapped in the format documented in
+ * fs/btrfs/lzo.c. For writes, the compression page size must match the
+ * filesystem page size.
+ */
+#define BTRFS_ENCODED_IO_COMPRESSION_LZO_4K 3
+#define BTRFS_ENCODED_IO_COMPRESSION_LZO_8K 4
+#define BTRFS_ENCODED_IO_COMPRESSION_LZO_16K 5
+#define BTRFS_ENCODED_IO_COMPRESSION_LZO_32K 6
+#define BTRFS_ENCODED_IO_COMPRESSION_LZO_64K 7
+#define BTRFS_ENCODED_IO_COMPRESSION_TYPES 8
nit: Make those an enum ? Same applies for the kernel counterpart patch.
On Thu, Oct 21, 2021 at 03:44:52PM +0300, Nikolay Borisov wrote:
On 1.09.21 г. 20:00, Omar Sandoval wrote:
quoted
From: Omar Sandoval <redacted>
Currently, we only create ordered extents when ram_bytes == num_bytes
and offset == 0. However, RWF_ENCODED writes may create extents which
Change RWF_ENCODED to simply encoded as we no longer rely on RWF flags,
same thing for the changelog in the next patch.
Oops, I checked the diff for stray references to RWF_ENCODED but I
forgot to check the commit messages. Thanks, I'll fix it.
On Thu, Oct 21, 2021 at 05:21:00PM +0300, Nikolay Borisov wrote:
On 1.09.21 г. 20:01, Omar Sandoval wrote:
quoted
From: Boris Burkov <redacted>
Send stream v2 can emit fallocate commands, so receive must support them
as well. The implementation simply passes along the arguments to the
syscall. Note that mode is encoded as a u32 in send stream but fallocate
takes an int, so there is a unsigned->signed conversion there.
Signed-off-by: Boris Burkov <redacted>
Reviewed-by: Nikolay Borisov <redacted>
However, kernel support for this hasn't landed, the kernel counterpart
patches add definitions but don't actually implement the code. BY the
looks of it it would seem that the proper send stream versioning could
be added first before any of this code lands. In this case we can simply
have the encoded writes stuff as protocol version 2 and leave the rest
of the commands for v3 for example.
The original idea for this was to minimize protocol revisions. This way,
when we get around to implementing fallocate on the send side, we
wouldn't need another update on the receive side. I still like that idea
since the receive side is so trivial.
From: Nikolay Borisov <hidden> Date: 2021-10-22 06:35:41
On 1.09.21 г. 20:01, Omar Sandoval wrote:
From: Boris Burkov <redacted>
To make the btrfs send ioctl use the stream v2 format requires passing
BTRFS_SEND_FLAG_STREAM_V2 in flags. Further, to cause the ioctl to emit
encoded_write commands for encoded extents, we must set that flag as
well as BTRFS_SEND_FLAG_COMPRESSED. Finally, we bump up the version in
send.h as well, since we are now fully compatible with v2.
Add two command line arguments to btrfs send: --stream-version and
--compressed-data. --stream-version requires an argument which it parses
as an integer and sets STREAM_V2 if the argument is 2. --compressed-data
does not require an argument and automatically implies STREAM_V2 as well
(COMPRESSED alone causes the ioctl to error out).
Some examples to illustrate edge cases:
// v1, old format and no encoded_writes
btrfs send subvol
btrfs send --stream-version 1 subvol
// v2 and compressed, we will see encoded_writes
btrfs send --compressed-data subvol
btrfs send --compressed-data --stream-version 2 subvol
// v2 only, new format but no encoded_writes
btrfs send --stream-version 2 subvol
// error: compressed needs version >= 2
btrfs send --compressed-data --stream-version 1 subvol
// error: invalid version (not 1 or 2)
btrfs send --stream-version 3 subvol
btrfs send --compressed-data --stream-version 0 subvol
btrfs send --compressed-data --stream-version 10 subvol
Why would we want to predicate the compressed writes usage on anything
other than the stream version?
@@ -55,7 +55,21 @@ send in 'NO_FILE_DATA' mode The output stream does not contain any file data and thus cannot be used to transfer changes. This mode is faster and is useful to show the differences in metadata.--q|--quiet::::++--stream-version <1|2>::+Use the given send stream version. The default is 1. Version 2 encodes file+data slightly more efficiently; it is also required for sending compressed data+directly (see '--compressed-data'). Version 2 requires at least btrfs-progs+5.12 on both the sender and receiver and at least Linux 5.12 on the sender.+
The version of progs needs to be adjusted but I assume this will be done
by David when this patchset is merged as we don't have a target ATM.
quoted hunk
+--compressed-data::
+Send data that is compressed on the filesystem directly without decompressing
+it. If the receiver supports encoded I/O (see `encoded_io`(7)), it can also
+write it directly without decompressing it. Otherwise, the receiver will fall
+back to decompressing it and writing it normally. This implies
+'--stream-version 2'.
+
+-q|--quiet::
(deprecated) alias for global '-q' option
-v|--verbose::
(deprecated) alias for global '-v' option
@@ -452,6 +452,21 @@ static const char * const cmd_send_usage[] = {" does not contain any file data and thus cannot be used"," to transfer changes. This mode is faster and useful to"," show the differences in metadata.",+"--stream-version <1|2>",+" Use the given send stream version. The default is",+" 1. Version 2 encodes file data slightly more",+" efficiently; it is also required for sending",+" compressed data directly (see --compressed-data).",+" Version 2 requires at least btrfs-progs 5.12 on both",+" the sender and receiver and at least Linux 5.12 on the",+" sender.",
On Wed, Sep 01, 2021 at 10:00:56AM -0700, Omar Sandoval wrote:
From: Omar Sandoval <redacted>
I'm adding Btrfs ioctls to read and write compressed data, and rather
than duplicating the checks in rw_verify_area(), let's just export it.
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Omar Sandoval <redacted>
---
fs/internal.h | 5 -----
fs/read_write.c | 1 +
include/linux/fs.h | 1 +
3 files changed, 2 insertions(+), 5 deletions(-)
Could I please get an ack from the VFS side on this patch and "fs:
export variant of generic_write_checks without iov_iter"? We're going
the route of doing this as a Btrfs ioctl since we couldn't agree on a
generic interface, so this is all I need from the VFS.
On Thu, Nov 18, 2021 at 11:19:39AM -0800, Omar Sandoval wrote:
On Wed, Sep 01, 2021 at 10:00:56AM -0700, Omar Sandoval wrote:
quoted
From: Omar Sandoval <redacted>
I'm adding Btrfs ioctls to read and write compressed data, and rather
than duplicating the checks in rw_verify_area(), let's just export it.
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Omar Sandoval <redacted>
---
fs/internal.h | 5 -----
fs/read_write.c | 1 +
include/linux/fs.h | 1 +
3 files changed, 2 insertions(+), 5 deletions(-)
Could I please get an ack from the VFS side on this patch and "fs:
export variant of generic_write_checks without iov_iter"? We're going
the route of doing this as a Btrfs ioctl since we couldn't agree on a
generic interface, so this is all I need from the VFS.