Shared file pages are never accounted in memory overcommit code,
so it isn't reasonable to count them in a code that limits the
maximal size of a process in OVERCOMMIT_NONE mode.

If a process has few large file mappings, the consequent attempts
to allocate anonymous memory may unexpectedly fail with -ENOMEM,
while there is free memory and overcommit limit if significantly
larger than the committed amount (as displayed in /proc/meminfo).

The problem is significantly smoothed by commit c9b1d0981fcc
("mm: limit growth of 3% hardcoded other user reserve"),
which limits the impact of this check with 128Mb (tunable via sysctl),
but it can still be a problem on small machines.

Signed-off-by: Roman Gushchin <redacted>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andrew Shewmaker <redacted>
Cc: Rik van Riel <redacted>
Cc: Konstantin Khlebnikov <redacted>
---
 mm/mmap.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/mmap.c b/mm/mmap.c
index 7f684d5..151fadf 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c

@@ -220,7 +220,7 @@ int __vm_enough_memory(struct mm_struct *mm, long pages, int cap_sys_admin)
 	 */
 	if (mm) {
 		reserve = sysctl_user_reserve_kbytes >> (PAGE_SHIFT - 10);
-		allowed -= min(mm->total_vm / 32, reserve);
+		allowed -= min((mm->total_vm - mm->shared_vm) / 32, reserve);
 	}
 
 	if (percpu_counter_read_positive(&vm_committed_as) < allowed)

-- 
2.1.0