[PATCH ipvs-next v3 0/2] sched: Add cond_resched_rcu_lock() helper

Add a helper that for use in loops which read data protected by RCU and
may have a large number of iterations.  Such an example is dumping the list
of connections known to IPVS: ip_vs_conn_array() and ip_vs_conn_seq_next().

This series also updates IPVS to use the helper.

Simon Horman (2):
  sched: add cond_resched_rcu() helper
  ipvs: use cond_resched_rcu() helper when walking connections

 include/linux/sched.h           |  9 +++++++++
 net/netfilter/ipvs/ip_vs_conn.c | 23 ++++++++---------------
 2 files changed, 17 insertions(+), 15 deletions(-)

-- 
1.8.2.1

This is intended for use in loops which read data protected by RCU and may
have a large number of iterations.  Such an example is dumping the list of
connections known to IPVS: ip_vs_conn_array() and ip_vs_conn_seq_next().

The benefits are for CONFIG_PREEMPT_RCU=y where we save CPU cycles
by moving rcu_read_lock and rcu_read_unlock out of large loops
but still allowing the current task to be preempted after every
loop iteration for the CONFIG_PREEMPT_RCU=n case.

The call to cond_resched() is not needed when CONFIG_PREEMPT_RCU=y.
Thanks to Paul E. McKenney for explaining this and for the
final version that checks the context with CONFIG_DEBUG_ATOMIC_SLEEP=y
for all possible configurations.

The function can be empty in the CONFIG_PREEMPT_RCU case,
rcu_read_lock and rcu_read_unlock are not needed in this case
because the task can be preempted on indication from scheduler.
Thanks to Peter Zijlstra for catching this and for his help
in trying a solution that changes __might_sleep.

Initial cond_resched_rcu_lock() function suggested by Eric Dumazet.

Tested-by: Julian Anastasov <ja@ssi.bg>
Signed-off-by: Julian Anastasov <ja@ssi.bg>
Signed-off-by: Simon Horman <horms@verge.net.au>
---
 include/linux/sched.h | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index e692a02..2080446 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h

@@ -2608,6 +2608,15 @@ extern int __cond_resched_softirq(void);
 	__cond_resched_softirq();					\
 })
 
+static inline void cond_resched_rcu(void)
+{
+#if defined(CONFIG_DEBUG_ATOMIC_SLEEP) || !defined(CONFIG_PREEMPT_RCU)
+	rcu_read_unlock();
+	cond_resched();
+	rcu_read_lock();
+#endif
+}
+
 /*
  * Does a critical section need to be broken due to another
  * task waiting?: (technically does not depend on CONFIG_PREEMPT,

-- 
1.8.2.1

This avoids the situation where walking of a large number of connections
may prevent scheduling for a long time while also avoiding excessive
calls to rcu_read_unlock() and rcu_read_lock().

Note that in the case of !CONFIG_PREEMPT_RCU this will
add a call to cond_resched().

[ja@ssi.bg: add cond_resched_rcu to more places]
Signed-off-by: Julian Anastasov <ja@ssi.bg>
Signed-off-by: Simon Horman <horms@verge.net.au>
---
 net/netfilter/ipvs/ip_vs_conn.c | 23 ++++++++---------------
 1 file changed, 8 insertions(+), 15 deletions(-)

diff --git a/net/netfilter/ipvs/ip_vs_conn.c b/net/netfilter/ipvs/ip_vs_conn.c
index a083bda..c8c52a9 100644
--- a/net/netfilter/ipvs/ip_vs_conn.c
+++ b/net/netfilter/ipvs/ip_vs_conn.c

@@ -975,8 +975,7 @@ static void *ip_vs_conn_array(struct seq_file *seq, loff_t pos)
 				return cp;
 			}
 		}
-		rcu_read_unlock();
-		rcu_read_lock();
+		cond_resched_rcu();
 	}
 
 	return NULL;

@@ -1015,8 +1014,7 @@ static void *ip_vs_conn_seq_next(struct seq_file *seq, void *v, loff_t *pos)
 			iter->l = &ip_vs_conn_tab[idx];
 			return cp;
 		}
-		rcu_read_unlock();
-		rcu_read_lock();
+		cond_resched_rcu();
 	}
 	iter->l = NULL;
 	return NULL;

@@ -1206,17 +1204,13 @@ void ip_vs_random_dropentry(struct net *net)
 	int idx;
 	struct ip_vs_conn *cp, *cp_c;
 
+	rcu_read_lock();
 	/*
 	 * Randomly scan 1/32 of the whole table every second
 	 */
 	for (idx = 0; idx < (ip_vs_conn_tab_size>>5); idx++) {
 		unsigned int hash = net_random() & ip_vs_conn_tab_mask;
 
-		/*
-		 *  Lock is actually needed in this loop.
-		 */
-		rcu_read_lock();
-
 		hlist_for_each_entry_rcu(cp, &ip_vs_conn_tab[hash], c_list) {
 			if (cp->flags & IP_VS_CONN_F_TEMPLATE)
 				/* connection template */

@@ -1252,8 +1246,9 @@ void ip_vs_random_dropentry(struct net *net)
 				__ip_vs_conn_put(cp);
 			}
 		}
-		rcu_read_unlock();
+		cond_resched_rcu();
 	}
+	rcu_read_unlock();
 }
 
 

@@ -1267,11 +1262,8 @@ static void ip_vs_conn_flush(struct net *net)
 	struct netns_ipvs *ipvs = net_ipvs(net);
 
 flush_again:
+	rcu_read_lock();
 	for (idx = 0; idx < ip_vs_conn_tab_size; idx++) {
-		/*
-		 *  Lock is actually needed in this loop.
-		 */
-		rcu_read_lock();
 
 		hlist_for_each_entry_rcu(cp, &ip_vs_conn_tab[idx], c_list) {
 			if (!ip_vs_conn_net_eq(cp, net))

@@ -1286,8 +1278,9 @@ flush_again:
 				__ip_vs_conn_put(cp);
 			}
 		}
-		rcu_read_unlock();
+		cond_resched_rcu();
 	}
+	rcu_read_unlock();
 
 	/* the counter may be not NULL, because maybe some conn entries
 	   are run by slow timer handler or unhashed but still referred */

-- 
1.8.2.1

On Wed, May 22, 2013 at 02:50:30PM +0900, Simon Horman wrote:

Add a helper that for use in loops which read data protected by RCU and
may have a large number of iterations.  Such an example is dumping the list
of connections known to IPVS: ip_vs_conn_array() and ip_vs_conn_seq_next().

This series also updates IPVS to use the helper.

Simon Horman (2):
  sched: add cond_resched_rcu() helper
  ipvs: use cond_resched_rcu() helper when walking connections

 include/linux/sched.h           |  9 +++++++++
 net/netfilter/ipvs/ip_vs_conn.c | 23 ++++++++---------------
 2 files changed, 17 insertions(+), 15 deletions(-)

Acked-by: Peter Zijlstra <peterz@infradead.org>

How do you want to go about merging this? Do you want to route it
through the net tree or through -tip?

From: Peter Zijlstra <peterz@infradead.org>
Date: Wed, 22 May 2013 09:54:38 +0200

On Wed, May 22, 2013 at 02:50:30PM +0900, Simon Horman wrote:

quoted

Add a helper that for use in loops which read data protected by RCU and
may have a large number of iterations.  Such an example is dumping the list
of connections known to IPVS: ip_vs_conn_array() and ip_vs_conn_seq_next().

This series also updates IPVS to use the helper.

Simon Horman (2):
  sched: add cond_resched_rcu() helper
  ipvs: use cond_resched_rcu() helper when walking connections

 include/linux/sched.h           |  9 +++++++++
 net/netfilter/ipvs/ip_vs_conn.c | 23 ++++++++---------------
 2 files changed, 17 insertions(+), 15 deletions(-)

Acked-by: Peter Zijlstra <peterz@infradead.org>

How do you want to go about merging this? Do you want to route it
through the net tree or through -tip?

I think we can have Pablo merge it into his netfilter tree.

On Wed, May 22, 2013 at 01:31:08AM -0700, David Miller wrote:

From: Peter Zijlstra <peterz@infradead.org>
Date: Wed, 22 May 2013 09:54:38 +0200

quoted

On Wed, May 22, 2013 at 02:50:30PM +0900, Simon Horman wrote:

quoted

Add a helper that for use in loops which read data protected by RCU and
may have a large number of iterations.  Such an example is dumping the list
of connections known to IPVS: ip_vs_conn_array() and ip_vs_conn_seq_next().

This series also updates IPVS to use the helper.

Simon Horman (2):
  sched: add cond_resched_rcu() helper
  ipvs: use cond_resched_rcu() helper when walking connections

 include/linux/sched.h           |  9 +++++++++
 net/netfilter/ipvs/ip_vs_conn.c | 23 ++++++++---------------
 2 files changed, 17 insertions(+), 15 deletions(-)

Acked-by: Peter Zijlstra <peterz@infradead.org>

How do you want to go about merging this? Do you want to route it
through the net tree or through -tip?

I think we can have Pablo merge it into his netfilter tree.

Works for me. Thanks!

On Wed, May 22, 2013 at 11:00:51AM +0200, Peter Zijlstra wrote:

On Wed, May 22, 2013 at 01:31:08AM -0700, David Miller wrote:

quoted

From: Peter Zijlstra <peterz@infradead.org>
Date: Wed, 22 May 2013 09:54:38 +0200

quoted

On Wed, May 22, 2013 at 02:50:30PM +0900, Simon Horman wrote:

quoted

Add a helper that for use in loops which read data protected by RCU and
may have a large number of iterations.  Such an example is dumping the list
of connections known to IPVS: ip_vs_conn_array() and ip_vs_conn_seq_next().

This series also updates IPVS to use the helper.

Simon Horman (2):
  sched: add cond_resched_rcu() helper
  ipvs: use cond_resched_rcu() helper when walking connections

 include/linux/sched.h           |  9 +++++++++
 net/netfilter/ipvs/ip_vs_conn.c | 23 ++++++++---------------
 2 files changed, 17 insertions(+), 15 deletions(-)

Acked-by: Peter Zijlstra <peterz@infradead.org>

How do you want to go about merging this? Do you want to route it
through the net tree or through -tip?

I think we can have Pablo merge it into his netfilter tree.

Works for me. Thanks!

Thanks, I will queue it up in ipvs-next and pass it to Pablo from there.

On Wed, May 22, 2013 at 02:50:31PM +0900, Simon Horman wrote:

This is intended for use in loops which read data protected by RCU and may
have a large number of iterations.  Such an example is dumping the list of
connections known to IPVS: ip_vs_conn_array() and ip_vs_conn_seq_next().

The benefits are for CONFIG_PREEMPT_RCU=y where we save CPU cycles
by moving rcu_read_lock and rcu_read_unlock out of large loops
but still allowing the current task to be preempted after every
loop iteration for the CONFIG_PREEMPT_RCU=n case.

The call to cond_resched() is not needed when CONFIG_PREEMPT_RCU=y.
Thanks to Paul E. McKenney for explaining this and for the
final version that checks the context with CONFIG_DEBUG_ATOMIC_SLEEP=y
for all possible configurations.

The function can be empty in the CONFIG_PREEMPT_RCU case,
rcu_read_lock and rcu_read_unlock are not needed in this case
because the task can be preempted on indication from scheduler.
Thanks to Peter Zijlstra for catching this and for his help
in trying a solution that changes __might_sleep.

Initial cond_resched_rcu_lock() function suggested by Eric Dumazet.

Applied, thanks.

On Wed, May 22, 2013 at 02:50:32PM +0900, Simon Horman wrote:

This avoids the situation where walking of a large number of connections
may prevent scheduling for a long time while also avoiding excessive
calls to rcu_read_unlock() and rcu_read_lock().

Note that in the case of !CONFIG_PREEMPT_RCU this will
add a call to cond_resched().

Applied, thanks.

On Wed, May 22, 2013 at 01:31:08AM -0700, David Miller wrote:

From: Peter Zijlstra <peterz@infradead.org>
Date: Wed, 22 May 2013 09:54:38 +0200

quoted

On Wed, May 22, 2013 at 02:50:30PM +0900, Simon Horman wrote:

quoted

Add a helper that for use in loops which read data protected by RCU and
may have a large number of iterations.  Such an example is dumping the list
of connections known to IPVS: ip_vs_conn_array() and ip_vs_conn_seq_next().

This series also updates IPVS to use the helper.

Simon Horman (2):
  sched: add cond_resched_rcu() helper
  ipvs: use cond_resched_rcu() helper when walking connections

 include/linux/sched.h           |  9 +++++++++
 net/netfilter/ipvs/ip_vs_conn.c | 23 ++++++++---------------
 2 files changed, 17 insertions(+), 15 deletions(-)

Acked-by: Peter Zijlstra <peterz@infradead.org>

How do you want to go about merging this? Do you want to route it
through the net tree or through -tip?

I think we can have Pablo merge it into his netfilter tree.

I did so. Thanks David.

* David Miller [off-list ref] wrote:

From: Peter Zijlstra <peterz@infradead.org>
Date: Wed, 22 May 2013 09:54:38 +0200

quoted

On Wed, May 22, 2013 at 02:50:30PM +0900, Simon Horman wrote:

quoted

Add a helper that for use in loops which read data protected by RCU and
may have a large number of iterations.  Such an example is dumping the list
of connections known to IPVS: ip_vs_conn_array() and ip_vs_conn_seq_next().

This series also updates IPVS to use the helper.

Simon Horman (2):
  sched: add cond_resched_rcu() helper
  ipvs: use cond_resched_rcu() helper when walking connections

 include/linux/sched.h           |  9 +++++++++
 net/netfilter/ipvs/ip_vs_conn.c | 23 ++++++++---------------
 2 files changed, 17 insertions(+), 15 deletions(-)

Acked-by: Peter Zijlstra <peterz@infradead.org>

How do you want to go about merging this? Do you want to route it
through the net tree or through -tip?

I think we can have Pablo merge it into his netfilter tree.

Yeah, that's what we agreed upon earlier during the review of these 
patches.

Thanks,

	Ingo