This series allows to selectively enable/disable polling for completions in the
block layer on a per-I/O basis.  For this it resurrects the
preadv2/pwritev2 syscalls that Milosz prepared a while ago (and which are
much simpler now due to VFS changes that happened in the meantime).
That approach also had a man page update prepared, which I will resubmit
with the current flags once this series makes it in.

Polling for block I/O is important to reduce the latency on flash and post-flash
storage technologies.  On the fastest NVMe controller I have access to it
almost halves latencies from over 7 microseconds to about 4 microseonds.
But it only is usesful if we actually care for the latency of this particular I/O,
and generally is a waste if enabled for all I/O to a given device.  This series
uses the per-I/O flags in preadv2/pwritev2 to control this behavior.  The
alternative would be a new O_* flag set at open time or using fcntl, but this is
still to corse-grained for some applications and we're starting to run out out
of open flags.

Thanks Christoph for re-submitting this. I for one am very supportive of being able to set priority (and other) flags on a per IO basis. I did some testing of this on a NVMe SSD that uses DRAM rather than NAND as its backing store. My performance absolutes are a bit worse than yours but the improvement of HIPRI over a normal IO was about the same with one thread (3-4us) and was a little bit more (6-7us) at a higher thread count. 

I used a fork of fio with a (rather ugly) hack to enable the new syscalls [1]. I then tested this on a per-thread basis using the following simple shell script. 

#!/bin/bash

FIO=/home/mtr/batesste/fio-git/fio
FILE=/dev/nvme0n1
THREADS=5
TIME=30

$FIO --name lowpri --filename=$FILE --size=1G --direct=1 \
    --ioengine=pvsync --rw=randread --bs=4k --runtime=$TIME \
    --numjobs=$THREADS --iodepth=1 --randrepeat=0 \
    --refill_buffers \
    --name hipri --filename=$FILE --size=1G --direct=1 \
    --ioengine=pv2sync --rw=randread --bs=4k --runtime=$TIME \
    --numjobs=$THREADS --iodepth=1 --randrepeat=0 \
    --refill_buffers

I also reviewed the code and it all looks good to me!

For the series:

Reviewed-by: Stephen Bates <redacted>
Tested-by: Stephen Bates <redacted> 

Note that there are plenty of other use cases for preadv2/pwritev2 as well,
but I'd like to concentrate on this one for now.  Example are: non-blocking
reads (the original purpose), per-I/O O_SYNC, user space support for T10
DIF/DIX applications tags and probably some more.

 Totally agree!

[1] https://github.com/sbates130272/fio/tree/hipri