For statistics, the same everything except layout:

Offset: around 700-780MB/sec
created: mdadm --create /dev/md3 --run -b none --level=10 --layout=o2
--chunk=16 --raid-devices=4 /dev/nvme0n1 /dev/nvme4n1 /dev/nvme3n1
/dev/nvme5n1

md3 : active raid10 nvme5n1[3] nvme3n1[2] nvme4n1[1] nvme0n1[0]
      7501212288 blocks super 1.2 16K chunks 2 offset-copies [4/4] [UUUU]
      [>....................]  resync =  1.5% (119689152/7501212288)
finish=156.3min speed=786749K/sec

near:around 700MB/sec
created: mdadm --create /dev/md3 --run -b none --level=10 --layout=n2
--chunk=16 --raid-devices=4 /dev/nvme0n1 /dev/nvme4n1 /dev/nvme3n1
/dev/nvme5n1

md3 : active raid10 nvme5n1[3] nvme3n1[2] nvme4n1[1] nvme0n1[0]
      7501212320 blocks super 1.2 16K chunks 2 near-copies [4/4] [UUUU]
      [>....................]  resync =  0.5% (42373104/7501212320)
finish=175.7min speed=707262K/sec

On Sat, Sep 2, 2023 at 3:23 AM CoolCold [off-list ref] wrote:

Good day!

I have 4 NVMe new drives which are planned to replace 2 current NVMe
drives, serving primarily as MYSQL storage, Hetzner dedicated server
AX161 if it matters. Drives are SAMSUNG MZQL23T8HCLS-00A07, 3.8TB .
System - Ubuntu 20.04 / 5.4.0-153-generic #170-Ubuntu

So the strange thing I do observe, is its initial raid sync speed.
Created with:
mdadm --create /dev/md3 --run -b none --level=10 --layout=f2
--chunk=16 --raid-devices=4 /dev/nvme0n1 /dev/nvme4n1 /dev/nvme3n1
/dev/nvme5n1

sync speed:

md3 : active raid10 nvme5n1[3] nvme3n1[2] nvme4n1[1] nvme0n1[0]
      7501212288 blocks super 1.2 16K chunks 2 far-copies [4/4] [UUUU]
      [=>...................]  resync =  6.2% (466905632/7501212288)
finish=207.7min speed=564418K/sec

If I try to create RAID1 with just two drives - sync speed is around
3.2GByte per second, sysclt is tuned of course:
dev.raid.speed_limit_max = 8000000

Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5]
[raid4] [raid10]
md70 : active raid1 nvme4n1[1] nvme5n1[0]
      3750606144 blocks super 1.2 [2/2] [UU]
      [>....................]  resync =  1.5% (58270272/3750606144)
finish=19.0min speed=3237244K/sec

From iostat, drives are basically doing just READs, no writes.
Quick tests with fio, mounting single drive shows it can do around 30k
IOPS with 16kb ( fio --rw=write --ioengine=sync --fdatasync=1
--directory=test-data --size=8200m --bs=16k --name=mytest ) so likely
issue are not drives themselves.

Not sure where to look further, please advise.

--
Best regards,
[COOLCOLD-RIPN]

-- 
Best regards,
[COOLCOLD-RIPN]