Re: [PATCH] btrfs: scrub: per-device bandwidth control

David Sterba wrote:

Add sysfs interface to limit io during scrub. We relied on the ionice
interface to do that, eg. the idle class let the system usable while
scrub was running. This has changed when mq-deadline got widespread and
did not implement the scheduling classes. That was a CFQ thing that got
deleted. We've got numerous complaints from users about degraded
performance.

Currently only BFQ supports that but it's not a common scheduler and we
can't ask everybody to switch to it.

Alternatively the cgroup io limiting can be used but that also a
non-trivial setup (v2 required, the controller must be enabled on the
system). This can still be used if desired.

Other ideas that have been explored: piggy-back on ionice (that is set
per-process and is accessible) and interpret the class and classdata as
bandwidth limits, but this does not have enough flexibility as there are
only 8 allowed and we'd have to map fixed limits to each value. Also
adjusting the value would need to lookup the process that currently runs
scrub on the given device, and the value is not sticky so would have to
be adjusted each time scrub runs.

Running out of options, sysfs does not look that bad:

- it's accessible from scripts, or udev rules
- the name is similar to what MD-RAID has
   (/proc/sys/dev/raid/speed_limit_max or /sys/block/mdX/md/sync_speed_max)
- the value is sticky at least for filesystem mount time
- adjusting the value has immediate effect
- sysfs is available in constrained environments (eg. system rescue)
- the limit also applies to device replace

Sysfs:

- raw value is in bytes
- values written to the file accept suffixes like K, M
- file is in the per-device directory /sys/fs/btrfs/FSID/devinfo/DEVID/scrub_speed_max
- 0 means use default priority of IO

Being just the average user I ponder if this feature is an absolute 
hard-limit or not? E.g. would this limit be applied all the time even if 
the filesystem is idle? or is it dynamic in the sense that scrub can run 
a full speed if the filesystem is (reasonably) idle.

If this is or could somehow be dynamic would it be possible to set the 
value to something like for example 30M:15M which may be interpreted as 
30MiB/s when not idle and 15MiB/s when not idle.

I myself would actually much prefer if it was possible to set a reserved 
bandwidth for all non-scrub related stuff. for example maybe up to max 
75% of the bandwidth always reserved for non-scrub related read/writes.
This would allow the scrub to run at full speed if no other I/O is is 
taking place in which case the scrub would be limited to 25%.