Re: [patch 7/11] net: Use bigrefs for net_device.refcount
From: Eric Dumazet <hidden>
Date: 2005-09-13 18:28:07
Also in:
lkml
Ravikiran G Thirumalai a écrit :
The net_device has a refcnt used to keep track of it's uses. This is used at the time of unregistering the network device (module unloading ..) (see netdev_wait_allrefs) . For loopback_dev , this refcnt increment/decrement is causing unnecessary traffic on the interlink for NUMA system affecting it's performance. This patch improves tbench numbers by 6% on a 8way x86 Xeon (x445).
===================================================================
quoted hunk ↗ jump to hunk
--- alloc_percpu-2.6.13.orig/include/linux/netdevice.h 2005-08-28 16:41:01.000000000 -0700 +++ alloc_percpu-2.6.13/include/linux/netdevice.h 2005-09-12 11:54:21.000000000 -0700@@ -37,6 +37,7 @@ #include <linux/config.h> #include <linux/device.h> #include <linux/percpu.h> +#include <linux/bigref.h> struct divert_blk; struct vlan_group;@@ -377,7 +378,7 @@ /* device queue lock */ spinlock_t queue_lock; /* Number of references to this device */ - atomic_t refcnt; + struct bigref netdev_refcnt; /* delayed register/unregister */ struct list_head todo_list; /* device name hash chain */@@ -677,11 +678,11 @@
Hum... Did you tried to place refcnt/netdev_refcnt in a separate cache line than queue_lock ? I got good results too... > /* device queue lock */ > spinlock_t queue_lock; > /* Number of references to this device */ > - atomic_t refcnt; > + struct bigref netdev_refcnt ____cacheline_aligned_in_smp ; > /* delayed register/unregister */ > struct list_head todo_list; > /* device name hash chain */ Every time a cpu take the queue_lock spinlock, it exclusively gets one cache line. If another cpu try to access netdev_refcnt, it has to grab this cache line (even if properely per_cpu designed, there is still one shared field). In fact the whole struct net_device should be re-ordered for SMP/NUMA performance. Eric