Hi Pintu,

I am also looking for a mailing list where we can share memory related
issues at kernel level. I tried subscribing to that mailing list but it
says Invalid ID :

quoted

quoted

quoted

--001a11436926db8726053c49ac67

**** Command '--001a11436926db8726053c49ac67' not recognized.

quoted

quoted

quoted

Content-Type: text/plain; charset=UTF-8

**** Command 'content-type:' not recognized.

quoted

quoted

quoted

subscribe linux-mm@kvack.org

**** subscribe: unknown list 'linux-mm@kvack.org'.

Thanks

On Mon, Sep 12, 2016 at 11:42 AM, PINTU KUMAR [off-list ref] wrote:

Dear Ankur,

I would suggest you register to linux-mm@kvack.org and explain your
issues in details.
There are other experts here, who can guide you.

Few comments are inline below.

quoted

From: Ankur Tank [mailto:Ankur.Tank@LntTechservices.com]
Sent: Saturday, September 10, 2016 5:26 PM
To: pintu.k@samsung.com
Cc: artfri2@gmail.com
Subject: Memory fragmentation issue related suggestion request

Hello Pintukumar,

TL;DR
We have an issue in our Linux box, what looks like memory fragmentation

issue,

quoted

while searching on net I referred talk you gave in Embedded Linux Conf.

I have several talks in ELC, not sure which one you are referring to.
Please point out.

quoted

I am facing this issue for couple of weeks so thought to ask you for

suggestions.

quoted

Please forgive me If I offended you by writing mail to you, Ignore mail

if you feel so.

quoted

Details
We are facing one issue in our Embedded Linux board, Our board is

Beaglebone

quoted

black based custom board, with 4GB eMMC as storage. We are using Linux

kernel

quoted

3.12.

In addition, you may need to provide the following information:
RAM size ?
cat /proc/meminfo  (before and after the operation)
cat /proc/buddyinfo (before and after the operation)
cat /proc/vmstat (before and after the operation)

quoted

Our firmware upgrade strategy is using backgup partition for Bootloader,

Kernel,

quoted

dtb, rootfs.
So,
During firmware upgrade with big rootfs and running dd to read the

partition in raw

quoted

mode.
In short looks like those operations are overloading the system.

I am not sure, but I think this is the crude way of taking the backup.
This will certainly overload your system.
FOTA upgrade experts can give more comments here.

quoted

From below log looks like pages above 32KB size is not available and may

be

quoted

because of that rootfs tar on the emmc is failing.
I have following queries in that regards,

1.       Do you think it is a memory fragmentation ?

Yes, if all above 32KB (2^3 order) pages are not available, and pages are
available in lower orders (2^0/1/2) then its certainly fragmentation
problem.
However, as I said, you need to provide the following output to confirm:
cat /proc/buddyinfo

quoted

May be silly to ask so but just to confirm, because I had added the

software swap

quoted

however with that also we were seeing issue reproducible and swap was

not full at

quoted

that time ☹

Well, adding swap should help a bit but it may not solve the problem
completely.
How much swap did you actually allocated?
What kind of swap you used ?
Is it ZRAM/ZSWAP (with compression support) ?
What is the swappiness ratio ? (/proc/sys/vm/swappiness)

quoted

2.       If it is so how do we handle it ? is there a some way similar

to your shrinker

quoted

utility to reclaim the memory pages ?

Not sure which shrinker utility are you referring to ?
Is it : /proc/sys/vm/shrink_memory ?

quoted

Any suggestion would help me move forward,

Did you tried enabling CONFIG_COMPACTION ?
Try using ZRAM or ZSWAP (~30% of MemTotal).
Try tuning : /proc/sys/vm/dirty_{background_ratio/bytes} and others.
[Refer kernel/documentation for the same]

From the logs, I observed the following:

quoted

[ 6676.674219] mmcqd/1: page allocation failure: order:1, mode:0x200020

Order-1 allocation is failing, so pages might be sitting in order-0.

quoted

[ 6676.674739]  free_cma:1982

You have around ~7MB of CMA free pages, so this cannot be used for
non-movable allocation.

quoted

[ 6676.674885] 51661 total pagecache pages

You have huge amount of memory sitting in caches. These can be reclaimed
in back ground (with slight performance degradation).
To experiment and debug you can try: echo 3 > /proc/sys/vm/drop_caches

quoted

[ 6676.674925] Total swap = 0kB

Swap is not enabled on your system.

quoted

Regards,
Ankur

Error log
----------------------------

[ 6676.674219] mmcqd/1: page allocation failure: order:1, mode:0x200020
   [ 6676.674256] CPU: 0 PID: 612 Comm: mmcqd/1 Tainted: P           O

3.12.10-005-

quoted

ts-armv7l #2
    [ 6676.674321] [<c0012d24>] (unwind_backtrace+0x0/0xf4) from

[<c0011130>]

quoted

(show_stack+0x10/0x14)
    [ 6676.674355] [<c0011130>] (show_stack+0x10/0x14) from [<c0087548>]
(warn_alloc_failed+0xe0/0x118)
    [ 6676.674383] [<c0087548>] (warn_alloc_failed+0xe0/0x118) from

[<c008a3ac>]

quoted

(__alloc_pages_nodemask+0x74c/0x8f8)
    [ 6676.674413] [<c008a3ac>] (__alloc_pages_nodemask+0x74c/0x8f8)

from

quoted

[<c00b2e8c>] (cache_alloc_refill+0x328/0x620)
    [ 6676.674436] [<c00b2e8c>] (cache_alloc_refill+0x328/0x620) from
[<c00b3224>] (__kmalloc+0xa0/0xe8)
    [ 6676.674471] [<c00b3224>] (__kmalloc+0xa0/0xe8) from [<c0212904>]
(edma_prep_slave_sg+0x84/0x388)
    [ 6676.674505] [<c0212904>] (edma_prep_slave_sg+0x84/0x388) from
[<c02ec0a0>] (omap_hsmmc_request+0x414/0x508)
    [ 6676.674544] [<c02ec0a0>] (omap_hsmmc_request+0x414/0x508) from
[<c02d6748>] (mmc_start_request+0xc4/0xe0)
    [ 6676.674568] [<c02d6748>] (mmc_start_request+0xc4/0xe0) from
[<c02d7530>] (mmc_start_req+0x2d8/0x38c)
    [ 6676.674589] [<c02d7530>] (mmc_start_req+0x2d8/0x38c) from

[<c02e4818>]

quoted

(mmc_blk_issue_rw_rq+0xb4/0x9d8)
    [ 6676.674611] [<c02e4818>] (mmc_blk_issue_rw_rq+0xb4/0x9d8) from
[<c02e52e0>] (mmc_blk_issue_rq+0x1a4/0x468)
    [ 6676.674631] [<c02e52e0>] (mmc_blk_issue_rq+0x1a4/0x468) from
[<c02e5c68>] (mmc_queue_thread+0x88/0x118)
    [ 6676.674657] [<c02e5c68>] (mmc_queue_thread+0x88/0x118) from
[<c004d8b8>] (kthread+0xb4/0xb8)
    [ 6676.674681] [<c004d8b8>] (kthread+0xb4/0xb8) from [<c000e298>]
(ret_from_fork+0x14/0x3c)
    [ 6676.674691] Mem-info:
    [ 6676.674700] Normal per-cpu:
    [ 6676.674711] CPU    0: hi:   90, btch:  15 usd:  79
    [ 6676.674739] active_anon:4889 inactive_anon:13 isolated_anon:0
    [ 6676.674739]  active_file:8082 inactive_file:43196 isolated_file:0
    [ 6676.674739]  unevictable:422 dirty:2 writeback:1152 unstable:0
    [ 6676.674739]  free:3286 slab_reclaimable:1090

slab_unreclaimable:915

quoted

    [ 6676.674739]  mapped:1593 shmem:39 pagetables:181 bounce:0
    [ 6676.674739]  free_cma:1982
    [ 6676.674800] Normal free:13144kB min:2004kB low:2504kB high:3004kB
active_anon:19556kB inactive_anon:52kB active_file:32328kB
inactive_file:172784kB unevictable:o
    [ 6676.674813] lowmem_reserve[]: 0 0 0
    [ 6676.674831] Normal: 2584*4kB (UMC) 217*8kB (C) 57*16kB (C) 5*32kB

(C)

quoted

0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB =
13144kB
    [ 6676.674885] 51661 total pagecache pages
    [ 6676.674900] 0 pages in swap cache
    [ 6676.674910] Swap cache stats: add 0, delete 0, find 0/0
    [ 6676.674918] Free swap  = 0kB
    [ 6676.674925] Total swap = 0kB
    [ 6676.674938] SLAB: Unable to allocate memory on node 0 (gfp=0x20)
    [ 6676.674949]   cache: kmalloc-8192, object size: 8192, order: 1
    [ 6676.674962]   node 0: slabs: 3/3, objs: 3/3, free: 0
    [ 6676.674984] omap_hsmmc 481d8000.mmc: prep_slave_sg() failed
    [ 6676.674997] omap_hsmmc 481d8000.mmc: MMC start dma failure
    [ 6676.676181] mmcblk0: unknown error -1 sending read/write command,

card

quoted

status 0x900
    [ 6676.676300] end_request: I/O error, dev mmcblk0, sector 27648
    [ 6676.676318] Buffer I/O error on device mmcblk0p9, logical block

896

quoted

    [ 6676.676329] lost page write due to I/O error on mmcblk0p9
    [ 6676.676401] end_request: I/O error, dev mmcblk0, sector 27656
    [ 6676.676415] Buffer I/O error on device mmcblk0p9, logical block

897

quoted

    [ 6676.676425] lost page write due to I/O error on mmcblk0p9
    [ 6676.676450] end_request: I/O error, dev mmcblk0, sector 27664
    [ 6676.676461] Buffer I/O error on device mmcblk0p9, logical block

898

quoted

    [ 6676.676471] lost page write due to I/O error on mmcblk0p9
    [ 6676.676494] end_request: I/O error, dev mmcblk0, sector 27672
    [ 6676.676505] Buffer I/O error on device mmcblk0p9, logical block

899

quoted

    [ 6676.676515] lost page write due to I/O error on mmcblk0p9
    [ 6676.676537] end_request: I/O error, dev mmcblk0, sector 27680
    [ 6676.676548] Buffer I/O error on device mmcblk0p9, logical block

900

quoted

    [ 6676.676558] lost page write due to I/O error on mmcblk0p9
    [ 6676.676580] end_request: I/O error, dev mmcblk0, sector 27688
    [ 6676.676591] Buffer I/O error on device mmcblk0p9, logical block

901

quoted

    [ 6676.676601] lost page write due to I/O error on mmcblk0p9
    [ 6676.676622] end_request: I/O error, dev mmcblk0, sector 27696
    [ 6676.676634] Buffer I/O error on device mmcblk0p9, logical block

902

quoted

    [ 6676.676643] lost page write due to I/O error on mmcblk0p9
    [ 6676.676665] end_request: I/O error, dev mmcblk0, sector 27704
    [ 6676.676676] Buffer I/O error on device mmcblk0p9, logical block

903

quoted

    [ 6676.676685] lost page write due to I/O error on mmcblk0p9
    [ 6676.676707] end_request: I/O error, dev mmcblk0, sector 27712
    [ 6676.676718] Buffer I/O error on device mmcblk0p9, logical block

904

quoted

    [ 6676.676728] lost page write due to I/O error on mmcblk0p9
    [ 6676.676749] end_request: I/O error, dev mmcblk0, sector 27720
    [ 6676.678266] mmcqd/1: page allocation failure: order:1,

mode:0x200020

quoted

    [ 6676.678285] CPU: 0 PID: 612 Comm: mmcqd/1 Tainted: P           O

3.12.10-005-

quoted

ts-armv7l #2
    [ 6676.678330] [<c0012d24>] (unwind_backtrace+0x0/0xf4) from

[<c0011130>]

quoted

(show_stack+0x10/0x14)
    [ 6676.678358] [<c0011130>] (show_stack+0x10/0x14) from [<c0087548>]
(warn_alloc_failed+0xe0/0x118)
    [ 6676.678385] [<c0087548>] (warn_alloc_failed+0xe0/0x118) from

[<c008a3ac>]

quoted

(__alloc_pages_nodemask+0x74c/0x8f8)
    [ 6676.678412] [<c008a3ac>] (__alloc_pages_nodemask+0x74c/0x8f8)

from

quoted

[<c00b2e8c>] (cache_alloc_refill+0x328/0x620)
    [ 6676.678434] [<c00b2e8c>] (cache_alloc_refill+0x328/0x620) from
[<c00b3224>] (__kmalloc+0xa0/0xe8)
    [ 6676.678464] [<c00b3224>] (__kmalloc+0xa0/0xe8) from [<c0212904>]
(edma_prep_slave_sg+0x84/0x388)
    [ 6676.678493] [<c0212904>] (edma_prep_slave_sg+0x84/0x388) from
[<c02ec0a0>] (omap_hsmmc_request+0x414/0x508)
    [ 6676.678524] [<c02ec0a0>] (omap_hsmmc_request+0x414/0x508) from
[<c02d6748>] (mmc_start_request+0xc4/0xe0)
    [ 6676.678547] [<c02d6748>] (mmc_start_request+0xc4/0xe0) from
[<c02d7530>] (mmc_start_req+0x2d8/0x38c)
    [ 6676.678568] [<c02d7530>] (mmc_start_req+0x2d8/0x38c) from

[<c02e4994>]

quoted

(mmc_blk_issue_rw_rq+0x230/0x9d8)
    [ 6676.678589] [<c02e4994>] (mmc_blk_issue_rw_rq+0x230/0x9d8) from
[<c02e52e0>] (mmc_blk_issue_rq+0x1a4/0x468)
    [ 6676.678608] [<c02e52e0>] (mmc_blk_issue_rq+0x1a4/0x468) from
[<c02e5c68>] (mmc_queue_thread+0x88/0x118)
    [ 6676.678632] [<c02e5c68>] (mmc_queue_thread+0x88/0x118) from
[<c004d8b8>] (kthread+0xb4/0xb8)
    [ 6676.678655] [<c004d8b8>] (kthread+0xb4/0xb8) from [<c000e298>]
(ret_from_fork+0x14/0x3c)
    [ 6676.678664] Mem-info: