Re: Regression in throughput between kvm guests over virtual bridge

On 2017年09月18日 11:13, Jason Wang wrote:

On 2017年09月16日 03:19, Matthew Rosato wrote:

quoted

quoted

It looks like vhost is slowed down for some reason which leads to more
idle time on 4.13+VHOST_RX_BATCH=1. Appreciated if you can collect the
perf.diff on host, one for rx and one for tx.

perf data below for the associated vhost threads, baseline=4.12,
delta1=4.13, delta2=4.13+VHOST_RX_BATCH=1

Client vhost:

60.12%  -11.11%  -12.34%  [kernel.vmlinux]   [k] raw_copy_from_user
13.76%   -1.28%   -0.74%  [kernel.vmlinux]   [k] get_page_from_freelist
  2.00%   +3.69%   +3.54%  [kernel.vmlinux]   [k] __wake_up_sync_key
  1.19%   +0.60%   +0.66%  [kernel.vmlinux]   [k] __alloc_pages_nodemask
  1.12%   +0.76%   +0.86%  [kernel.vmlinux]   [k] copy_page_from_iter
  1.09%   +0.28%   +0.35%  [vhost]            [k] vhost_get_vq_desc
  1.07%   +0.31%   +0.26%  [kernel.vmlinux]   [k] alloc_skb_with_frags
  0.94%   +0.42%   +0.65%  [kernel.vmlinux]   [k] alloc_pages_current
  0.91%   -0.19%   -0.18%  [kernel.vmlinux]   [k] memcpy
  0.88%   +0.26%   +0.30%  [kernel.vmlinux]   [k] __next_zones_zonelist
  0.85%   +0.05%   +0.12%  [kernel.vmlinux]   [k] iov_iter_advance
  0.79%   +0.09%   +0.19%  [vhost]            [k] __vhost_add_used_n
  0.74%                    [kernel.vmlinux]   [k] get_task_policy.part.7
  0.74%   -0.01%   -0.05%  [kernel.vmlinux]   [k] tun_net_xmit
  0.60%   +0.17%   +0.33%  [kernel.vmlinux]   [k] policy_nodemask
  0.58%   -0.15%   -0.12%  [ebtables]         [k] ebt_do_table
  0.52%   -0.25%   -0.22%  [kernel.vmlinux]   [k] __alloc_skb
    ...
  0.42%   +0.58%   +0.59%  [kernel.vmlinux]   [k] eventfd_signal
    ...
  0.32%   +0.96%   +0.93%  [kernel.vmlinux]   [k] finish_task_switch
    ...
          +1.50%   +1.16%  [kernel.vmlinux]   [k] get_task_policy.part.9
          +0.40%   +0.42%  [kernel.vmlinux]   [k] __skb_get_hash_symmetr
          +0.39%   +0.40%  [kernel.vmlinux]   [k] _copy_from_iter_full
          +0.24%   +0.23%  [vhost_net]        [k] vhost_net_buf_peek

Server vhost:

61.93%  -10.72%  -10.91%  [kernel.vmlinux]   [k] raw_copy_to_user
  9.25%   +0.47%   +0.86%  [kernel.vmlinux]   [k] free_hot_cold_page
  5.16%   +1.41%   +1.57%  [vhost]            [k] vhost_get_vq_desc
  5.12%   -3.81%   -3.78%  [kernel.vmlinux]   [k] skb_release_data
  3.30%   +0.42%   +0.55%  [kernel.vmlinux]   [k] raw_copy_from_user
  1.29%   +2.20%   +2.28%  [kernel.vmlinux]   [k] copy_page_to_iter
  1.24%   +1.65%   +0.45%  [vhost_net]        [k] handle_rx
  1.08%   +3.03%   +2.85%  [kernel.vmlinux]   [k] __wake_up_sync_key
  0.96%   +0.70%   +1.10%  [vhost]            [k] translate_desc
  0.69%   -0.20%   -0.22%  [kernel.vmlinux]   [k] tun_do_read.part.10
  0.69%                    [kernel.vmlinux]   [k] tun_peek_len
  0.67%   +0.75%   +0.78%  [kernel.vmlinux]   [k] eventfd_signal
  0.52%   +0.96%   +0.98%  [kernel.vmlinux]   [k] finish_task_switch
  0.50%   +0.05%   +0.09%  [vhost]            [k] vhost_add_used_n
    ...
          +0.63%   +0.58%  [vhost_net]        [k] vhost_net_buf_peek
          +0.32%   +0.32%  [kernel.vmlinux]   [k] _copy_to_iter
          +0.19%   +0.19%  [kernel.vmlinux]   [k] __skb_get_hash_symmetr
          +0.11%   +0.21%  [vhost]            [k] vhost_umem_interval_tr

Looks like for some unknown reason which leads more wakeups.

Could you please try to attached patch to see if it solves or mitigate 
the issue?

Thanks 

My bad, please try this.

Thanks