On 2018-02-12 11:39, Ellis H. Wilson III wrote:

On 02/12/2018 11:02 AM, Austin S. Hemmelgarn wrote:

quoted

BTRFS in general works fine at that scale, dependent of course on the 
level of concurrent access you need to support.  Each tree update 
needs to lock a bunch of things in the tree itself, and having large 
numbers of clients writing to the same set of files concurrently can 
cause lock contention issues because of this, especially if all of 
them are calling fsync() or fdatasync() regularly.  These issues can 
be mitigated by segregating workloads into their own subvolumes (each 
subvolume is a mostly independent filesystem tree), but it sounds like 
you're already doing that, so I don't think that would be an issue for 
you.

Hmm...I'll think harder about this.  There is potential for us to 
artificially divide access to files across subvolumes automatically 
because of the way we are using BTRFS as a backing store for our 
parallel file system.  So far even with around 1000 threads across about 
10 machines accessing BTRFS via our parallel filesystem over the wire 
we've not seen issues, but if we do I have some ways out I've not 
explored yet.  Thanks!

For what it's worth, most of the issues I've personally seen with 
parallel performance involved very heavy use of fsync(), or lots of 
parallel calls to stat() and statvfs() happening while files are also 
being written to, so it may just be the way you happen to be doing 
things just doesn't cause issues.