On 21 Sep 2021, at 16:04, Peter Xu [off-list ref] wrote:

Hi, Tiberiu,

On Tue, Sep 21, 2021 at 08:52:32AM +0000, Tiberiu Georgescu wrote:

quoted

I tested it some more, and it still looks like the mincore() syscall considers pages
in the swap cache as "in memory". This is how I tested:

1. Create a cgroup with 1M limit_in_bytes, and allow swapping
2. mmap 1024 pages (both shared and private present the same behaviour)
3. write to all pages in order
4. compare mincore output with pagemap output

This is an example of a usual mincore output in this scenario, shortened for
coherency (4x8 instead of 16x64):
00000000
00000000
00001110   <- this bugs me
01111111

The last 7 bits are definitely marking pages present in memory, but there are
some other bits set a little earlier. When comparing this output with the pagemap,
indeed, there are 7 consecutive pages present, and the rest of them are
swapped, including those 3 which are marked as present by mincore.
At this point, I can only assume the bits in between are on the swap cache.

If you have another explanation, please share it with me. In the meanwhile,
I will rework the doc patch, and see if there is any other way to differentiate
clearly between the three types of pages. If not, I guess we'll stick to
mincore() and a best-effort 5th step.

IIUC it could be because of that the pages are still in swap cache, so
mincore() will return 1 for them too.

That is my assumption as well.

What swap device are you using?  I'm wildly guessing you're not using frontswap
like zram.  If that's the case, would you try zram?  That should flush the page
synchronously iiuc, then all the "suspecious 1s" will go away above.

Correct, I was not using frontswap.

To do that, you may need to firstly turn off your current swap:

       # swapoff -a

Then to configure zram you need:

       # modprobe zram
       # echo 4G > /sys/block/zram0/disksize
       # mkswap --label zram0 /dev/zram0
       # swapon --priority 100 /dev/zram0

Quotting from here:

       https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.archlinux.org_title_Improving-5Fperformance-23zram-5For-5Fzswap&d=DwIBaQ&c=s883GpUCOChKOHiocYtGcg&r=rRM5dtWOv0DNo5dDxZ2U16jl4WAw6ql5szOKa9cu_RA&m=XWzLVqSSl8CSEcw2x6sUmspJhiUJei2gq6GTiaky8hk&s=k3BDgO9LN63Nn3vxorlc41MlUYzOUN0efajz4lol-k8&e= 

Then you can try run the same test program again.

Thanks, it worked!

Hmmm, so if we put emphasis on the accuracy of swap info, or accuracy in
general, we need to use frontswap. Otherwise, mincore() could suffer from
race conditions, and mark pages in the swap cache as being present.

Do you reckon this info (frontswap for mincore) should be present in
the pagemap docs? I wouldn't want to bloat the section either.

Kind regards,
Tibi