On 08/01/16 13:09, Michal Hocko wrote:

On Mon 01-08-16 12:52:40, Ralf-Peter Rohbeck wrote:

quoted

On 01.08.2016 12:43, Michal Hocko wrote:

quoted

On Mon 01-08-16 12:35:51, Ralf-Peter Rohbeck wrote:

quoted

On 01.08.2016 12:26, Michal Hocko wrote:

[...]

quoted

quoted

the amount of dirty pages is much smaller as well as the anonymous
memory. The biggest portion seems to be in the page cache. The memory

The page cache will always be full if I'm writing at full steam to multiple
drives, no?

Yes, the memory full of page cache is not unusual. The large portion of
that memory being dirty/writeback can be a problem. That is why we have
a dirty memory throttling which slows down (throttles) writers to keep
the amount reasonable. What is your dirty throttling setup?
$ grep . /proc/sys/vm/dirty*

and what is your storage setup?

root@fs:~# grep . /proc/sys/vm/dirty*
/proc/sys/vm/dirty_background_bytes:0
/proc/sys/vm/dirty_background_ratio:10
/proc/sys/vm/dirty_bytes:0
/proc/sys/vm/dirty_expire_centisecs:3000
/proc/sys/vm/dirty_ratio:20

With your 8G of RAM this can be quite a lot of dirty data at once. Is
your storage able to write that back in a reasonable time? I mean this
shouldn't cause the OOM killer but it can lead to some unexpected stalls
especially when there are a lot of writers AFAIU. dirty_bytes knob
should help to define a better cap.

The main filesystems are on the MegaRAID and can do 500-600 MB/s. 
Writing to the USB drives only pushes about 90MB/s per drive.

quoted

/proc/sys/vm/dirtytime_expire_seconds:43200
/proc/sys/vm/dirty_writeback_centisecs:500


Storage setup:

root@fs:~# lsscsi
[0:2:0:0]    disk    LSI      MR9271-8iCC      3.29  /dev/sda
[0:2:1:0]    disk    LSI      MR9271-8iCC      3.29  /dev/sdb
[9:0:0:0]    disk    TOSHIBA  External USB 3.0 5438  /dev/sdf
[10:0:0:0]   disk    Seagate  Backup+ Desk     050B  /dev/sdc
[11:0:0:0]   disk    Seagate  Expansion Desk   9400  /dev/sdd
[12:0:0:0]   disk    Seagate  Backup+ Desk     050B /dev/sde
[13:0:0:0]   disk    Seagate  Expansion Desk   9400 /dev/sdg
[14:0:0:0]   disk    TOSHIBA  External USB 3.0 5438 /dev/sdl
[15:0:0:0]   disk    Seagate  Expansion Desk   9400 /dev/sdh
[16:0:0:0]   disk    Seagate  Expansion Desk   9400 /dev/sdi
[17:0:0:0]   disk    TOSHIBA  External USB 3.0 5438 /dev/sdm
[18:0:0:0]   disk    Seagate  Expansion Desk   9400 /dev/sdj
[19:0:0:0]   disk    Seagate  Expansion Desk   9400  /dev/sdk

sda is a 6x 1TB RAID5 and sdb is a single 480GB SSD, both on a MegaRAID
controller.

The rest are 4TB USB drives that I'm experimenting with.

Which devices did you write when hitting the OOM killer?

sdc, sdd and sde each at max speed, with a little bit of garden variety 
IO on sda and sdb.

----------------------------------------------------------------------
The information contained in this transmission may be confidential. Any disclosure, copying, or further distribution of confidential information is not permitted unless such privilege is explicitly granted in writing by Quantum. Quantum reserves the right to have electronic communications, including email and attachments, sent across its networks filtered through anti virus and spam software programs and retain such messages in order to comply with applicable data security and retention requirements. Quantum is not responsible for the proper and complete transmission of the substance of this communication or for any delay in its receipt.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>