Re: XFS Mount need ages

On Fri, Jun 25, 2021 at 02:05:29PM +0200, Ml Ml wrote:

After a loong time it mounted now. Here is some more info:

xfs_info /mnt/backup-cluster5
meta-data=/dev/rbd6              isize=512    agcount=65536, agsize=32768 blks

                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=0
data     =                       bsize=4096   blocks=2147483648, imaxpct=25
         =                       sunit=16     swidth=16 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2

                                                ^^^^^^^^^^^

         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

On Fri, Jun 25, 2021 at 12:49 PM Ml Ml [off-list ref] wrote:

quoted

Hello List,

i have a rbd block device with xfs on it. After resizing it (from 6TB
to 8TB i think) the mount need hours to complete:

I started the mount 15mins ago.:
  mount -nv /dev/rbd6 /mnt/backup-cluster5

ps:
root      1143  0.2  0.0   8904  3088 pts/0    D+   12:17   0:03  |
   \_ mount -nv /dev/rbd6 /mnt/backup-cluster5


There is no timeout or ANY msg in dmesg until now.

strace -p 1143  :  seems to do nothing.
iotop --pid=1143: uses about 50KB/sec

dd bs=1M count=2048 if=/dev/rbd6 of=/dev/null => gives me 50MB/sec


Any idea what´s the problem here?

You started with a ~10GB sized filesystem (based on sunit, agsize
and log size) and have grown it almost 3 orders of magnitude. This
is way beyond recommendations - problems with fs layout tend to
expose themselves before the filesystem has been grown by a factor
of 10x, let alone ~1000x.

What I'd say is happening here is that if mount needed to iterate
all the AGs for some reason (e.g. an unclean shutdown), and it now
has to read all the AGF and AGI headers.  That's 130,000 IOs it now
needs to do.  At 100 IOPS, that's going to take 1300 seconds - a bit
over 20 minutes. And mount may have to do more than this (read AG
btrees to count blocks), so it could be doing thousands of IOs per
AG, which because of the sub-optimal layout, is multiplied by tens
of thousands of times...

Ceph RBDs ain't the fastest devices around, so this is very likely.

When you deploy a TB scale device, start with a TB-scale device, not
a device that is <10GB. Starting with a TB scale device will give
you TB-scale AGs, not 128MB sized AGs. Scanning a couple of dozen
AGs takes very little time, scanning tens of thousands of tiny AGs
takes a long time...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com