Recently, I came across a similar issue with this. I was running a fio
command doing buffered IO on RAIO0 btrfs on SMR (zoned) devices and found
ignoring BTRFS_FS_CSUM_IMPL_FAST in should_async_write() improves the
performance.

I ran the same experiment on regular SSDs and on SMR HDDs, varying the
number of devices. In both case, increasing the number of devices results
in better performance for ignoring BTRFS_FS_CSUM_IMPL_FAST (disabling
inline csum).

Here is a detail I did.

Environment:
  CPU: 96 CPUs, 2 NUMA nodes
  mem: 1024GB

mkfs: mkfs.btrfs -d raid0 -m raid0 ${devs}

fio command:
fio --group_reporting --eta=always --eta-interval=10s --eta-newline=10s \
     --rw=write --fallocate=none \
     --direct=0 --ioengine=libaio --iodepth=32 --end_fsync=1 \
     --filesize=200G bs=$((64 * ${njobs}))k \
     --time_based --runtime=300s 
     --directory=/mnt --name=writer --numjobs=${njobs}

* Results on SSDs

njobs=6 for this setup.

- baseline: with inline checksum = misc-next
  - numdevs 1: WRITE: bw=442MiB/s (464MB/s), 442MiB/s-442MiB/s (464MB/s-464MB/s), io=302GiB (324GB), run=698553-698553msec
  - numdevs 2: WRITE: bw=873MiB/s (915MB/s), 873MiB/s-873MiB/s (915MB/s-915MB/s), io=425GiB (457GB), run=499052-499052msec
  - numdevs 3: WRITE: bw=1162MiB/s (1218MB/s), 1162MiB/s-1162MiB/s (1218MB/s-1218MB/s), io=491GiB (527GB), run=432872-432872msec
  - numdevs 4: WRITE: bw=1261MiB/s (1322MB/s), 1261MiB/s-1261MiB/s (1322MB/s-1322MB/s), io=493GiB (530GB), run=400496-400496msec
  - numdevs 5: WRITE: bw=1331MiB/s (1395MB/s), 1331MiB/s-1331MiB/s (1395MB/s-1395MB/s), io=494GiB (531GB), run=380279-380279msec
  - numdevs 6: WRITE: bw=1391MiB/s (1458MB/s), 1391MiB/s-1391MiB/s (1458MB/s-1458MB/s), io=499GiB (536GB), run=367312-367312msec

- without inline checksum
  - numdevs 1: WRITE: bw=437MiB/s (459MB/s), 437MiB/s-437MiB/s (459MB/s-459MB/s), io=299GiB (321GB), run=699787-699787msec
  - numdevs 2: WRITE: bw=850MiB/s (892MB/s), 850MiB/s-850MiB/s (892MB/s-892MB/s), io=419GiB (450GB), run=504463-504463msec
  - numdevs 3: WRITE: bw=1259MiB/s (1320MB/s), 1259MiB/s-1259MiB/s (1320MB/s-1320MB/s), io=539GiB (579GB), run=438666-438666msec
  - numdevs 4: WRITE: bw=1597MiB/s (1675MB/s), 1597MiB/s-1597MiB/s (1675MB/s-1675MB/s), io=636GiB (683GB), run=408050-408050msec
  - numdevs 5: WRITE: bw=2021MiB/s (2119MB/s), 2021MiB/s-2021MiB/s (2119MB/s-2119MB/s), io=759GiB (815GB), run=384534-384534msec
  - numdevs 6: WRITE: bw=2106MiB/s (2208MB/s), 2106MiB/s-2106MiB/s (2208MB/s-2208MB/s), io=760GiB (816GB), run=369705-369705msec

* Results on SMR HDDs (zoned mode)

njobs=30 for this setup

- baseline: with inline checksum = misc-next
  - numdevs  1: WRITE: bw=194MiB/s (204MB/s), 194MiB/s-194MiB/s (204MB/s-204MB/s), io=232GiB (249GB), run=1219953-1219953msec
  - numdevs  2: WRITE: bw=393MiB/s (412MB/s), 393MiB/s-393MiB/s (412MB/s-412MB/s), io=279GiB (299GB), run=725862-725862msec
  - numdevs  4: WRITE: bw=763MiB/s (800MB/s), 763MiB/s-763MiB/s (800MB/s-800MB/s), io=407GiB (437GB), run=545920-545920msec
  - numdevs  8: WRITE: bw=995MiB/s (1044MB/s), 995MiB/s-995MiB/s (1044MB/s-1044MB/s), io=448GiB (481GB), run=460426-460426msec
  - numdevs 16: WRITE: bw=1196MiB/s (1254MB/s), 1196MiB/s-1196MiB/s (1254MB/s-1254MB/s), io=447GiB (480GB), run=382588-382588msec
  - numdevs 20: WRITE: bw=1247MiB/s (1307MB/s), 1247MiB/s-1247MiB/s (1307MB/s-1307MB/s), io=445GiB (478GB), run=365526-365526msec
  - numdevs 25: WRITE: bw=1286MiB/s (1349MB/s), 1286MiB/s-1286MiB/s (1349MB/s-1349MB/s), io=443GiB (475GB), run=352474-352474msec
  - numdevs 30: WRITE: bw=1299MiB/s (1362MB/s), 1299MiB/s-1299MiB/s (1362MB/s-1362MB/s), io=437GiB (470GB), run=344890-344890msec

- without inline checksum
  - numdevs  1: WRITE: bw=219MiB/s (229MB/s), 219MiB/s-219MiB/s (229MB/s-229MB/s), io=236GiB (253GB), run=1103041-1103041msec
  - numdevs  2: WRITE: bw=426MiB/s (447MB/s), 426MiB/s-426MiB/s (447MB/s-447MB/s), io=297GiB (319GB), run=712921-712921msec
  - numdevs  4: WRITE: bw=734MiB/s (770MB/s), 734MiB/s-734MiB/s (770MB/s-770MB/s), io=383GiB (411GB), run=534348-534348msec
  - numdevs  8: WRITE: bw=1124MiB/s (1178MB/s), 1124MiB/s-1124MiB/s (1178MB/s-1178MB/s), io=497GiB (534GB), run=452973-452973msec
  - numdevs 16: WRITE: bw=1603MiB/s (1681MB/s), 1603MiB/s-1603MiB/s (1681MB/s-1681MB/s), io=652GiB (700GB), run=416390-416390msec
  - numdevs 20: WRITE: bw=1804MiB/s (1892MB/s), 1804MiB/s-1804MiB/s (1892MB/s-1892MB/s), io=705GiB (757GB), run=400029-400029msec
  - numdevs 25: WRITE: bw=1913MiB/s (2006MB/s), 1913MiB/s-1913MiB/s (2006MB/s-2006MB/s), io=660GiB (709GB), run=353584-353584msec
  - numdevs 30: WRITE: bw=1953MiB/s (2048MB/s), 1953MiB/s-1953MiB/s (2048MB/s-2048MB/s), io=658GiB (707GB), run=345082-345082msec

Also, I attached plots of ratio ("MiB/s without inline checksum" / "MiB/s
with inline checksum = misc-next") for these cases.

The plots shows disabling inline checksum can be better by at most 50% than
inline checksum, if we have many devices.

The result will vary depending on the environment, but the fast checksum
might not be fast enough if btrfs is served by RAID0 on multiple devices.

On Wed, Dec 06, 2023 at 03:22:19PM +0100, Linux regression tracking (Thorsten Leemhuis) wrote:

On 29.08.23 11:45, Linux regression tracking (Thorsten Leemhuis) wrote:

quoted

On 13.08.23 11:50, Wang Yugui wrote:

quoted

quoted

On 8/11/23 10:23 AM, Wang Yugui wrote:

quoted

quoted

On Wed, Aug 02, 2023 at 08:04:57AM +0800, Wang Yugui wrote:

quoted

quoted

And with only a revert of

"btrfs: submit IO synchronously for fast checksum implementations"?

GOOD performance when only (Revert "btrfs: submit IO synchronously for fast
checksum implementations") 

Ok, so you have a case where the offload for the checksumming generation
actually helps (by a lot).  Adding Chris to the Cc list as he was
involved with this.

quoted

quoted

quoted

-       if (test_bit(BTRFS_FS_CSUM_IMPL_FAST, &bbio->fs_info->flags))
+       if ((bbio->bio.bi_opf & REQ_META) && test_bit(BTRFS_FS_CSUM_IMPL_FAST, &bbio->fs_info->flags))
                return false;

This disables synchronous checksum calculation entirely for data I/O.

without this fix, data I/O checksum is always synchronous?
this is a feature change of "btrfs: submit IO synchronously for fast checksum implementations"?

It is never with the above patch.

quoted

quoted

Also I'm curious if you see any differents for a non-RAID0 (i.e.
single profile) workload.

'-m single -d single' is about 10% slow that '-m raid1 -d raid0' in this test
case.

How does it compare with and without the revert?  Can you add the numbers?

Looking through the thread, you're comparing -m single -d single, but
btrfs is still doing the raid.

Sorry to keep asking for more runs, but these numbers are a surprise,
and I probably won't have time today to reproduce before vacation next
week (sadly, Christoph and I aren't going together).

FWIW, seems this thread died down and the underlying reason for the
regression despite quite a bit of effort from the developers wasn't
found. Haven't noticed any similar reports either. Reverting apparently
is not a option. So in the end this afaics remains unfixed. In an ideal
"no regressions" world this shouldn't happen, but well, we sadly don't
live in one. So I'll stop tracking this issue, it's not worth the effort:

#regzbot inconclusive: despite some efforts from the developers could
not be fixed
#regzbot ignore-activity

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.