Hi Greg,
I saw this patch float past and thought of you... You may not be interested
any more, and it may be a perfectly good patch that does not need any
comment, but I thought I would let you know anyway.
Thanks Neil.
I've trimmed the cc list to limit the number of copies Trond and Bruce get:)
From: Eric Dumazet<redacted>
To: Trond Myklebust<redacted>
Cc: "J. Bruce Fields"<redacted>, Neil Brown<redacted>,
David Miller[off-list ref], linux-nfs@vger.kernel.org, netdev
[off-list ref], linux-kernel[off-list ref]
Subject: [PATCH] sunrpc: use better NUMA affinities
Use NUMA aware allocations to reduce latencies and increase throughput.
Briefly looking at the patch, it doesn't seem wrong but I'm surprised
it's (still) necessary.
Some years ago at SGI we encountered that same problem; we solved it by
delaying all the allocation of data structures associated with a thread
so that they were performed in the thread itself, after the thread had
been limited to run on a certain set of CPUs. Thus the thread's normal
allocation behaviour resulted in all of it's allocations being from
node-local pages. It was a pretty ugly patch, but it worked and made a
huge difference to NFS throughput on large NUMA boxes.
Later Jeff Layton converted the sunrpc svc startup code to use kthreads
and at the time I read his patches, pointed out this problem, and posted
my patch for comparison
http://linux-nfs.org/pipermail/nfsv4/2008-May/008760.html
I seem to remember coming to the conclusion that Jeff eventually
addressed this problem...am I misremembering or did something regress?
--
Greg.
From: Eric Dumazet <hidden> Date: 2011-07-29 06:30:43
Le vendredi 29 juillet 2011 à 16:05 +1000, Greg Banks a écrit :
On 29/07/11 15:32, NeilBrown wrote:
quoted
Hi Greg,
I saw this patch float past and thought of you... You may not be interested
any more, and it may be a perfectly good patch that does not need any
comment, but I thought I would let you know anyway.
Thanks Neil.
I've trimmed the cc list to limit the number of copies Trond and Bruce get:)
quoted
From: Eric Dumazet<redacted>
To: Trond Myklebust<redacted>
Cc: "J. Bruce Fields"<redacted>, Neil Brown<redacted>,
David Miller[off-list ref], linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, netdev
[off-list ref], linux-kernel[off-list ref]
Subject: [PATCH] sunrpc: use better NUMA affinities
Use NUMA aware allocations to reduce latencies and increase throughput.
Briefly looking at the patch, it doesn't seem wrong but I'm surprised
it's (still) necessary.
Some years ago at SGI we encountered that same problem; we solved it by
delaying all the allocation of data structures associated with a thread
so that they were performed in the thread itself, after the thread had
been limited to run on a certain set of CPUs. Thus the thread's normal
allocation behaviour resulted in all of it's allocations being from
node-local pages. It was a pretty ugly patch, but it worked and made a
huge difference to NFS throughput on large NUMA boxes.
Later Jeff Layton converted the sunrpc svc startup code to use kthreads
and at the time I read his patches, pointed out this problem, and posted
my patch for comparison
http://linux-nfs.org/pipermail/nfsv4/2008-May/008760.html
I seem to remember coming to the conclusion that Jeff eventually
addressed this problem...am I misremembering or did something regress?
Currently, all nfsd kthreads use memory for their kernel stack and
various initial data from a _single_ node, even if you use
sunrpc.pool_mode=pernode (or percpu)
With my patch, we make sure each thread gets its stack from its local
node.
Check commit 94dcf29a11b3d20a (kthread: use kthread_create_on_node()) to
see how this strategy already was adopted for ksoftirqd, kworker,
migration, and pktgend kthreads.
I only have small machines here (two nodes), so I cannot post
significative bench results, but it seems quite obvious to expect a good
increase.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Le vendredi 29 juillet 2011 à 16:05 +1000, Greg Banks a écrit :
quoted
On 29/07/11 15:32, NeilBrown wrote:
I seem to remember coming to the conclusion that Jeff eventually
addressed this problem...am I misremembering or did something regress?
Currently, all nfsd kthreads use memory for their kernel stack and
various initial data from a _single_ node, even if you use
sunrpc.pool_mode=pernode (or percpu)
That's just plain broken and I'm very pleased to see you fix it.
I was just surprised that it was still broken and wondering how that
happened. Looking at ToT I see that because I dropped the ball in 2008,
Jeff's patches didn't address the problem. In ToT
svc_pool_map_set_cpumask() is called *after* kthread_create() and
applies to the child thread, *after* it's stack has been allocated on
the wrong node. In the working SGI code, svc_pool_map_set_cpumask() is
called by the parent node on itself *before* calling kernel_thread() or
doing any of the data structure allocations, thus ensuring that
everything gets allocated using the default memory allocation policy,
which on SGI NFS servers was globally tuned to be "node-local".
With my patch, we make sure each thread gets its stack from its local
node.
Check commit 94dcf29a11b3d20a (kthread: use kthread_create_on_node()) to
see how this strategy already was adopted for ksoftirqd, kworker,
migration, and pktgend kthreads.
Ah, I see. It's unfortunate that the kthread_create() API ends up being
passed a CPU number but that's only used to format the name and not for
sensible things :(
--
Greg.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Christoph Hellwig <hch@infradead.org> Date: 2011-07-29 10:36:42
On Fri, Jul 29, 2011 at 04:53:21PM +1000, Greg Banks wrote:
quoted
Check commit 94dcf29a11b3d20a (kthread: use kthread_create_on_node()) to
see how this strategy already was adopted for ksoftirqd, kworker,
migration, and pktgend kthreads.
Ah, I see. It's unfortunate that the kthread_create() API ends up
being passed a CPU number but that's only used to format the name
and not for sensible things :(
kthread_create doesn't have a cpu argument - it has a printf-like format
string.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Sent from my iPhone
On 29/07/2011, at 20:36, Christoph Hellwig [off-list ref] wrote:
On Fri, Jul 29, 2011 at 04:53:21PM +1000, Greg Banks wrote:
quoted
quoted
Check commit 94dcf29a11b3d20a (kthread: use kthread_create_on_node
()) to
see how this strategy already was adopted for ksoftirqd, kworker,
migration, and pktgend kthreads.
Ah, I see. It's unfortunate that the kthread_create() API ends up
being passed a CPU number but that's only used to format the name
and not for sensible things :(
kthread_create doesn't have a cpu argument - it has a printf-like
format
string.
Sure, and a whole lot of the callsites are ("..._%d", cpu), hence the
unfortune :(
Greg.
From: Eric Dumazet <hidden> Date: 2011-07-29 12:11:37
Le vendredi 29 juillet 2011 à 21:58 +1000, Greg Banks a écrit :
Sure, and a whole lot of the callsites are ("..._%d", cpu), hence the
unfortune :(
BTW, we could name nfsd threads differently :
Currently, they all are named : "nfsd"
If SVC_POOL_PERCPU is selected, we could name them :
nfsd_c0 -> nfsd_cN
If SVC_POOL_PERNODE is selected, we could name them :
nfsd_n0 -> nfsd_nN
That would help to check with "ps aux" which cpu/nodes are under stress.
Sent from my iPhone
On 29/07/2011, at 22:11, Eric Dumazet [off-list ref] wrote:
Le vendredi 29 juillet 2011 à 21:58 +1000, Greg Banks a écrit :
quoted
Sure, and a whole lot of the callsites are ("..._%d", cpu), hence the
unfortune :(
BTW, we could name nfsd threads differently :
Currently, they all are named : "nfsd"
If SVC_POOL_PERCPU is selected, we could name them :
nfsd_c0 -> nfsd_cN
If SVC_POOL_PERNODE is selected, we could name them :
nfsd_n0 -> nfsd_nN
That would help to check with "ps aux" which cpu/nodes are under
stress.
I like it!
Greg.--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: J. Bruce Fields <hidden> Date: 2011-07-29 16:45:59
On Fri, Jul 29, 2011 at 04:53:21PM +1000, Greg Banks wrote:
On 29/07/11 16:30, Eric Dumazet wrote:
quoted
Le vendredi 29 juillet 2011 à 16:05 +1000, Greg Banks a écrit :
quoted
On 29/07/11 15:32, NeilBrown wrote:
I seem to remember coming to the conclusion that Jeff eventually
addressed this problem...am I misremembering or did something regress?
Currently, all nfsd kthreads use memory for their kernel stack and
various initial data from a _single_ node, even if you use
sunrpc.pool_mode=pernode (or percpu)
That's just plain broken and I'm very pleased to see you fix it.
Should I take that as a "Reviewed-by"?
I was just surprised that it was still broken and wondering how that
happened. Looking at ToT I see that because I dropped the ball in
2008, Jeff's patches didn't address the problem. In ToT
svc_pool_map_set_cpumask() is called *after* kthread_create() and
applies to the child thread, *after* it's stack has been allocated
on the wrong node. In the working SGI code,
svc_pool_map_set_cpumask() is called by the parent node on itself
*before* calling kernel_thread() or doing any of the data structure
allocations, thus ensuring that everything gets allocated using the
default memory allocation policy, which on SGI NFS servers was
globally tuned to be "node-local".
OK, so would it be enough to just move the svc_pool_map_set_cpumask()
back a few lines, or do we want Eric's approach, in order to have
something that will work better with other memory allocation policies?
--b.
quoted
With my patch, we make sure each thread gets its stack from its local
node.
Check commit 94dcf29a11b3d20a (kthread: use kthread_create_on_node()) to
see how this strategy already was adopted for ksoftirqd, kworker,
migration, and pktgend kthreads.
Ah, I see. It's unfortunate that the kthread_create() API ends up
being passed a CPU number but that's only used to format the name
and not for sensible things :(
--
Greg.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: J. Bruce Fields <hidden> Date: 2011-07-29 16:48:16
On Fri, Jul 29, 2011 at 04:05:40PM +1000, Greg Banks wrote:
On 29/07/11 15:32, NeilBrown wrote:
quoted
Hi Greg,
I saw this patch float past and thought of you... You may not be interested
any more, and it may be a perfectly good patch that does not need any
comment, but I thought I would let you know anyway.
Thanks Neil.
I've trimmed the cc list to limit the number of copies Trond and Bruce get:)
I appreciate the thought, but in future please don't trim cc's.
--b.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: J. Bruce Fields <hidden> Date: 2011-07-29 16:48:45
On Fri, Jul 29, 2011 at 11:30:05PM +1000, Greg Banks wrote:
Sent from my iPhone
On 29/07/2011, at 22:11, Eric Dumazet [off-list ref] wrote:
quoted
Le vendredi 29 juillet 2011 à 21:58 +1000, Greg Banks a écrit :
quoted
Sure, and a whole lot of the callsites are ("..._%d", cpu), hence the
unfortune :(
BTW, we could name nfsd threads differently :
Currently, they all are named : "nfsd"
If SVC_POOL_PERCPU is selected, we could name them :
nfsd_c0 -> nfsd_cN
If SVC_POOL_PERNODE is selected, we could name them :
nfsd_n0 -> nfsd_nN
That would help to check with "ps aux" which cpu/nodes are under
stress.
From: J. Bruce Fields <hidden> Date: 2011-07-29 16:53:52
On Fri, Jul 29, 2011 at 12:48:36PM -0400, bfields wrote:
On Fri, Jul 29, 2011 at 11:30:05PM +1000, Greg Banks wrote:
quoted
Sent from my iPhone
On 29/07/2011, at 22:11, Eric Dumazet [off-list ref] wrote:
quoted
Le vendredi 29 juillet 2011 à 21:58 +1000, Greg Banks a écrit :
quoted
Sure, and a whole lot of the callsites are ("..._%d", cpu), hence the
unfortune :(
BTW, we could name nfsd threads differently :
Currently, they all are named : "nfsd"
If SVC_POOL_PERCPU is selected, we could name them :
nfsd_c0 -> nfsd_cN
If SVC_POOL_PERNODE is selected, we could name them :
nfsd_n0 -> nfsd_nN
That would help to check with "ps aux" which cpu/nodes are under
stress.
I like it!
Yup, patch welcomed.--b.
(Annoying fact: some initscripts stop nfsd using a rough equivalent of
"killall nfsd". So the name of the threads is arguably ABI. I think
those initscripts are nuts and deserve what they get, but that may be
because I'm forgetting the reason they do that.)
--b.
From: Eric Dumazet <hidden> Date: 2011-07-29 18:15:09
Le vendredi 29 juillet 2011 à 12:53 -0400, J. Bruce Fields a écrit :
(Annoying fact: some initscripts stop nfsd using a rough equivalent of
"killall nfsd". So the name of the threads is arguably ABI. I think
those initscripts are nuts and deserve what they get, but that may be
because I'm forgetting the reason they do that.)
Yes, it seems we cant change then...
I tried on a RHEL4 distro and "service nfs stop" could not stop my nfsd
threads if I changed their names as described.
Next, I am going to try nfsd/...
By the way, any idea why after "service nfs stop" I cannot unload sunrpc
module ?
# lsmod
Module Size Used by
nfsd 285597 3
lockd 76103 1 nfsd
auth_rpcgss 45002 1 nfsd
sunrpc 231915 6 nfsd,lockd,auth_rpcgss
exportfs 3813 1 nfsd
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Sent from my iPhone
On 30/07/2011, at 2:45, "J. Bruce Fields" [off-list ref] wrote:
On Fri, Jul 29, 2011 at 04:53:21PM +1000, Greg Banks wrote:
quoted
On 29/07/11 16:30, Eric Dumazet wrote:
quoted
Le vendredi 29 juillet 2011 à 16:05 +1000, Greg Banks a écrit :
quoted
On 29/07/11 15:32, NeilBrown wrote:
I seem to remember coming to the conclusion that Jeff eventually
addressed this problem...am I misremembering or did something
regress?
Currently, all nfsd kthreads use memory for their kernel stack and
various initial data from a _single_ node, even if you use
sunrpc.pool_mode=pernode (or percpu)
That's just plain broken and I'm very pleased to see you fix it.
Should I take that as a "Reviewed-by"?
If you like.
quoted
[...] In ToT
svc_pool_map_set_cpumask() is called *after* kthread_create() and
applies to the child thread, *after* it's stack has been allocated
on the wrong node. In the working SGI code,
svc_pool_map_set_cpumask() is called by the parent node on itself
*before* calling kernel_thread() or doing any of the data structure
allocations, thus ensuring that everything gets allocated using the
default memory allocation policy, which on SGI NFS servers was
globally tuned to be "node-local".
OK, so would it be enough to just move the svc_pool_map_set_cpumask()
back a few lines, or do we want Eric's approach, in order to have
something that will work better with other memory allocation policies?
Relying on global policy was easy but not a great idea, I prefer
Eric's approach of doing it explicitly.
Greg.
Sent from my iPhone
On 30/07/2011, at 2:53, "J. Bruce Fields" [off-list ref] wrote:
On Fri, Jul 29, 2011 at 12:48:36PM -0400, bfields wrote:
quoted
On Fri, Jul 29, 2011 at 11:30:05PM +1000, Greg Banks wrote:
quoted
Sent from my iPhone
On 29/07/2011, at 22:11, Eric Dumazet [off-list ref]
wrote:
quoted
Le vendredi 29 juillet 2011 à 21:58 +1000, Greg Banks a écrit :
quoted
Sure, and a whole lot of the callsites are ("..._%d", cpu),
hence the
unfortune :(
BTW, we could name nfsd threads differently :
Currently, they all are named : "nfsd"
If SVC_POOL_PERCPU is selected, we could name them :
nfsd_c0 -> nfsd_cN
If SVC_POOL_PERNODE is selected, we could name them :
nfsd_n0 -> nfsd_nN
That would help to check with "ps aux" which cpu/nodes are under
stress.
I like it!
Yup, patch welcomed.--b.
(Annoying fact: some initscripts stop nfsd using a rough equivalent of
"killall nfsd". So the name of the threads is arguably ABI. I think
those initscripts are nuts and deserve what they get, but that may be
because I'm forgetting the reason they do that.)
We had this discussion in May-June 2008; it's because the nfsds were
once many many years ago userspace threads.
The "killall nfsd" semantics in those scripts are awful and lead to
problems shutting down when there are lots of threads. It would
probably be an improvement to provide a better shutdown mechanism and
force distros to use it.
Or, you could preserve the effective semantics by having a single
"nfsd" thread whose purpose is to notice that it's being signalled and
perform a clean shutdown (perhaps blocking the thread doing the kill()
call until the shutdown has completed).
Greg.--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Sent from my iPhone
On 30/07/2011, at 2:53, "J. Bruce Fields" [off-list ref] wrote:
quoted
On Fri, Jul 29, 2011 at 12:48:36PM -0400, bfields wrote:
quoted
On Fri, Jul 29, 2011 at 11:30:05PM +1000, Greg Banks wrote:
quoted
Sent from my iPhone
On 29/07/2011, at 22:11, Eric Dumazet [off-list ref]
wrote:
quoted
Le vendredi 29 juillet 2011 à 21:58 +1000, Greg Banks a écrit :
quoted
Sure, and a whole lot of the callsites are ("..._%d", cpu),
hence the
unfortune :(
BTW, we could name nfsd threads differently :
Currently, they all are named : "nfsd"
If SVC_POOL_PERCPU is selected, we could name them :
nfsd_c0 -> nfsd_cN
If SVC_POOL_PERNODE is selected, we could name them :
nfsd_n0 -> nfsd_nN
That would help to check with "ps aux" which cpu/nodes are under
stress.
I like it!
Yup, patch welcomed.--b.
(Annoying fact: some initscripts stop nfsd using a rough equivalent of
"killall nfsd". So the name of the threads is arguably ABI. I think
those initscripts are nuts and deserve what they get, but that may be
because I'm forgetting the reason they do that.)
We had this discussion in May-June 2008; it's because the nfsds were
once many many years ago userspace threads.
Even when they became kernel threads, 'kill' was the only way to kill them -
at first.
The "killall nfsd" semantics in those scripts are awful and lead to
problems shutting down when there are lots of threads. It would
probably be an improvement to provide a better shutdown mechanism and
force distros to use it.
rpc.nfsd 0
will stop all nfsd threads. Follow with
exportfs -f
and you should be done. I'm not 100% sure about the nfsv4 thread though -
would need to check.
And yes - I would love it if distros could standardise on start/stop scripts
so upstreams could maintain them sensibly. This is my personal number 1
reason for liking systemd - it pushes for this standardisation.
Or, you could preserve the effective semantics by having a single
"nfsd" thread whose purpose is to notice that it's being signalled and
perform a clean shutdown (perhaps blocking the thread doing the kill()
call until the shutdown has completed).
That's kinda neat. It would be an ugly wart to have to keep around, but
sometimes that the price we pay for "no regressions".
NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Sent from my iPhone
On 30/07/2011, at 2:53, "J. Bruce Fields" [off-list ref] wrote:
quoted
On Fri, Jul 29, 2011 at 12:48:36PM -0400, bfields wrote:
quoted
On Fri, Jul 29, 2011 at 11:30:05PM +1000, Greg Banks wrote:
quoted
Sent from my iPhone
On 29/07/2011, at 22:11, Eric Dumazet [off-list ref]
wrote:
quoted
Le vendredi 29 juillet 2011 à 21:58 +1000, Greg Banks a écrit :
quoted
Sure, and a whole lot of the callsites are ("..._%d", cpu),
hence the
unfortune :(
BTW, we could name nfsd threads differently :
Currently, they all are named : "nfsd"
If SVC_POOL_PERCPU is selected, we could name them :
nfsd_c0 -> nfsd_cN
If SVC_POOL_PERNODE is selected, we could name them :
nfsd_n0 -> nfsd_nN
That would help to check with "ps aux" which cpu/nodes are under
stress.
I like it!
Yup, patch welcomed.--b.
(Annoying fact: some initscripts stop nfsd using a rough equivalent of
"killall nfsd". So the name of the threads is arguably ABI. I think
those initscripts are nuts and deserve what they get, but that may be
because I'm forgetting the reason they do that.)
We had this discussion in May-June 2008; it's because the nfsds were
once many many years ago userspace threads.
Even when they became kernel threads, 'kill' was the only way to kill them -
at first.
quoted
The "killall nfsd" semantics in those scripts are awful and lead to
problems shutting down when there are lots of threads. It would
probably be an improvement to provide a better shutdown mechanism and
force distros to use it.
rpc.nfsd 0
will stop all nfsd threads. Follow with
exportfs -f
and you should be done. I'm not 100% sure about the nfsv4 thread though -
would need to check.
Should be fine.
--b.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Eric Dumazet <hidden> Date: 2011-07-30 04:08:41
Le vendredi 29 juillet 2011 à 19:48 -0400, J. Bruce Fields a écrit :
On Sat, Jul 30, 2011 at 09:30:25AM +1000, NeilBrown wrote:
quoted
rpc.nfsd 0
will stop all nfsd threads. Follow with
exportfs -f
and you should be done. I'm not 100% sure about the nfsv4 thread though -
would need to check.
Should be fine.
What are the needed steps to be able to unload nfsd module ?
So far, I am unable to perform this without using -f option.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sat, 30 Jul 2011 06:08:34 +0200 Eric Dumazet [off-list ref]
wrote:
Le vendredi 29 juillet 2011 à 19:48 -0400, J. Bruce Fields a écrit :
quoted
On Sat, Jul 30, 2011 at 09:30:25AM +1000, NeilBrown wrote:
quoted
rpc.nfsd 0
will stop all nfsd threads. Follow with
exportfs -f
and you should be done. I'm not 100% sure about the nfsv4 thread though -
would need to check.
Should be fine.
What are the needed steps to be able to unload nfsd module ?
So far, I am unable to perform this without using -f option.
Kill all nfsd threads.
This can be done with 'kill' or 'rpc.nfsd 0' or
echo 0 > /proc/fs/nfsd/threads
kill mountd
killall rpc.mountd
kill idmapd
unexport everything.
This can be done with 'exportfs -f' and checked by
grep . /proc/net/rpc/*/content
this should only contain comments
unmount the 'nfsd' filesystem
umount /proc/fs/nfsd
rmmod nfsd
(it look me ages to remember idmapd - but
lsof /proc/net/rpc/*/*
gave it away)
NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
The "killall nfsd" semantics in those scripts are awful and lead to
problems shutting down when there are lots of threads. It would
probably be an improvement to provide a better shutdown mechanism and
force distros to use it.
rpc.nfsd 0
will stop all nfsd threads. Follow with
exportfs -f
and you should be done. I'm not 100% sure about the nfsv4 thread though -
would need to check.
And yes - I would love it if distros could standardise on start/stop scripts
so upstreams could maintain them sensibly. This is my personal number 1
reason for liking systemd - it pushes for this standardisation.
I can't remember if this is getting fixed in Fedora--probably not. We
should fix it.
I'd also be fine with logging a deprecation message when someone signals
an nfsd thread.
quoted
Or, you could preserve the effective semantics by having a single
"nfsd" thread whose purpose is to notice that it's being signalled and
perform a clean shutdown (perhaps blocking the thread doing the kill()
call until the shutdown has completed).
That's kinda neat. It would be an ugly wart to have to keep around, but
sometimes that the price we pay for "no regressions".
Yep; patch welcomed.
--b.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: J. Bruce Fields <hidden> Date: 2011-08-27 00:02:28
On Sun, Jul 31, 2011 at 08:58:54AM +0200, Eric Dumazet wrote:
Le samedi 30 juillet 2011 à 08:23 +0200, Eric Dumazet a écrit :
quoted
Le samedi 30 juillet 2011 à 16:06 +1000, NeilBrown a écrit :
quoted
umount /proc/fs/nfsd
Thans a lot, this was the thing I missed !
Hmm, after this, I left things as the were (nfsd module loaded, but a 0
use count )
I got a BUG while updatedb was running,
(I tried this on a debian machine/kernel)
Same bug on :
cat /proc/self/mounts
Any luck figuring this out?
I tried reproducing, by stopping everything to get nfsd refcount to
zero:
# lsmod | grep '^nfsd'
nfsd 300025 0
then running 'cat /proc/self/mounts'.
But I don't see any crash.
This is only 3.1.0-rc1 plus some nfsd patches.
--b.
From: Eric Dumazet <hidden> Date: 2011-08-28 10:02:40
Le vendredi 26 août 2011 à 20:02 -0400, J. Bruce Fields a écrit :
On Sun, Jul 31, 2011 at 08:58:54AM +0200, Eric Dumazet wrote:
quoted
Le samedi 30 juillet 2011 à 08:23 +0200, Eric Dumazet a écrit :
quoted
Le samedi 30 juillet 2011 à 16:06 +1000, NeilBrown a écrit :
quoted
umount /proc/fs/nfsd
Thans a lot, this was the thing I missed !
Hmm, after this, I left things as the were (nfsd module loaded, but a 0
use count )
I got a BUG while updatedb was running,
(I tried this on a debian machine/kernel)
Same bug on :
cat /proc/self/mounts
Any luck figuring this out?
I tried reproducing, by stopping everything to get nfsd refcount to
zero:
# lsmod | grep '^nfsd'
nfsd 300025 0
then running 'cat /proc/self/mounts'.
But I don't see any crash.
This is only 3.1.0-rc1 plus some nfsd patches.
--b.
quoted
I'll take a look at this issue later, after my vacations, unless someone
wants to take a look before me ;)
Unfortunately, latest linux tree doesnt even boot on my machine, so I
cannot reproduce the problem right now.
Last message displayed being "Switching to clocksource tsc"
Previous kernels were continuing with :
[ 4.448417] 00:09: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[ 4.597727] Real Time Clock Driver v1.12b
Might be a problem with latest tty changes
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html