Re: client's caching of server-side capabilities

On Wed, 2021-06-30 at 12:48 -0400, Olga Kornievskaia wrote:

On Wed, Jun 30, 2021 at 11:23 AM J. Bruce Fields
[off-list ref] wrote:

quoted

On Tue, Jun 29, 2021 at 01:51:43PM +0000, Chuck Lever III wrote:

quoted

quoted

On Jun 29, 2021, at 9:48 AM, Olga Kornievskaia [off-list ref]
wrote:

On Tue, Jun 29, 2021 at 8:58 AM Chuck Lever III
[off-list ref] wrote:

quoted

quoted

On Jun 28, 2021, at 6:06 PM, Trond Myklebust
[off-list ref] wrote:

On Mon, 2021-06-28 at 16:23 -0400, Olga Kornievskaia wrote:

quoted

Hi folks,

I have a general question of why the client doesn't throw
away the
cached server's capabilities on server reboot. Say a
client mounted a
server when the server didn't support security_labels,
then the
server
was rebooted and support was enabled. Client re-
establishes its
clientid/session, recovers state, but assumes all the old
capabilities
apply. A remount is required to clear old/find new
capabilities. The
opposite is true that a capability could be removed (but
I'm assuming
that's a less practical example).

I'm curious what are the problems of clearing server
capabilities and
rediscovering them on reboot? Is it because a local
filesystem could
never have its attributes changed and thus a network file
system
can't
either?

Thank you.

In my opinion, the client should aim for the absolute
minimum overhead
on a server reboot. The goal should be to recover state and
get I/O
started again as quickly as possible.

I 100% agree with the above. However...

quoted

Detection of new features, etc
can wait until the client needs to restart.

A server reboot can be part of a failover to a different
server. I
think capability discovery needs to happen as part of server
reboot
recovery, it can't be optimized away.

Can you clarify what you mean by a "failover to a different
server"?

IP-based failover means that a server can crash, and its partner
can
detect that and take over the IP address and exports of the
failed
server. The replacement server doesn't have to have exactly the
same
set of capabilities.

So it could also lose capabilities?

Well, wouldn't the client lose capabilities even now? Operations
relying on those capabilities wouldn't work (ie., say security label
wouldn't be returned or an operation would error with ENOTSUPP). And
I
think when it comes to operations, that's fine as the capability
would
then be adjusted (removed).

To make it clear again, I'm not suggesting to do it at server reboot
as it was pointed out to cause performance problems.

Yep. The reason why I'd be more tolerant of this in the case of
migration/server failover is because in that case, the client is
already expected to trawl the various mountpoints for NFS4ERR_MOVED
errors, and running fs_locations probes anyway. The process is already
slow and disruptive, so throwing in an fsinfo probe to the new server
isn't really a big deal.

quoted

I'm a little nervous about server features being changed out from
under
the client while the client has the server mounted.

But, I don't know, looking quickly through the list of NFS_CAP_*
definitions in nfs_fs_sb.h, I'm not coming up with a case where we
couldn't handle it, maybe it's OK.

--b.

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com