From:   Eric Dumazet <edumazet@google.com>
Date:   Fri, 26 Aug 2022 08:17:25 -0700

On Thu, Aug 25, 2022 at 5:05 PM Kuniyuki Iwashima [off-list ref] wrote:

quoted

The more sockets we have in the hash table, the more time we spend
looking up the socket.  While running a number of small workloads on
the same host, they penalise each other and cause performance degradation.

Also, the root cause might be a single workload that consumes much more
resources than the others.  It often happens on a cloud service where
different workloads share the same computing resource.

On EC2 c5.24xlarge instance (196 GiB memory and 524288 (1Mi / 2) ehash
entries), after running iperf3 in different netns, creating 24Mi sockets
without data transfer in the root netns causes about 10% performance
regression for the iperf3's connection.

 thash_entries          sockets         length          Gbps
        524288                1              1          50.7
                           24Mi             48          45.1

It is basically related to the length of the list of each hash bucket.
For testing purposes to see how performance drops along the length,
I set 131072 (1Mi / 8) to thash_entries, and here's the result.

 thash_entries          sockets         length          Gbps
        131072                1              1          50.7
                            1Mi              8          49.9
                            2Mi             16          48.9
                            4Mi             32          47.3
                            8Mi             64          44.6
                           16Mi            128          40.6
                           24Mi            192          36.3
                           32Mi            256          32.5
                           40Mi            320          27.0
                           48Mi            384          25.0

To resolve the socket lookup degradation, we introduce an optional
per-netns hash table for TCP and UDP.  With a smaller hash table, we
can look up sockets faster and isolate noisy neighbours.  Also, we can
reduce lock contention.

We can control and check the hash size via sysctl knobs.  It requires
some tuning based on workloads, so the per-netns hash table is disabled
by default.

  # dmesg | cut -d ' ' -f 5- | grep "established hash"
  TCP established hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc hugepage)

  # sysctl net.ipv4.tcp_ehash_entries
  net.ipv4.tcp_ehash_entries = 524288  # can be changed by thash_entries

  # sysctl net.ipv4.tcp_child_ehash_entries
  net.ipv4.tcp_child_ehash_entries = 0  # disabled by default

  # ip netns add test1
  # ip netns exec test1 sysctl net.ipv4.tcp_ehash_entries
  net.ipv4.tcp_ehash_entries = -524288  # share the global ehash

  # sysctl -w net.ipv4.tcp_child_ehash_entries=100
  net.ipv4.tcp_child_ehash_entries = 100

  # sysctl net.ipv4.tcp_child_ehash_entries
  net.ipv4.tcp_child_ehash_entries = 128  # rounded up to 2^n

  # ip netns add test2
  # ip netns exec test2 sysctl net.ipv4.tcp_ehash_entries
  net.ipv4.tcp_ehash_entries = 128  # own per-netns ehash

  [ UDP has the same interface as udp_hash_entries and
    udp_child_hash_entries. ]

When creating per-netns concurrently with different sizes, we can
guarantee the size by doing one of these ways.

  1) Share the global hash table and create per-netns one

  First, unshare() with tcp_child_ehash_entries==0.  It creates dedicated
  netns sysctl knobs where we can safely change tcp_child_ehash_entries
  and clone()/unshare() to create a per-netns hash table.

  2) Lock the sysctl knob

This is orthogonal.

Your series should have been split in three really.

I do not want to discuss the merit of re-instating LOCK_MAND :/

I see.
I'll drop the flock() part at once and respin TCP part only in v2.