I do something with fewer moving parts for my backups:
dstring=`date +%Y%m%d`
DIRDATE=`date +%Y%m%d`

MONTH=`date +%Y%m`

DAY=`date +%d`
DIR=/2TB-backup
mkdir -p ${DIR}/backup/${MONTH}/${DIRDATE}

/usr/bin/rsync -xab --backup-dir=/${DIR}/backup/${MONTH}/${DIRDATE}/
<listofdirstobcakup> --excludefile <excludefilename> ${DIR} >>
${DIR}/backup/backups-${dstring}.out

If you run this every day/so often then you end up with any file that
is changed being in the backup/month/day structure and can see all
file changes going back as far as you have enough space for.

And when you are running low on space you delete the directories
associated with older dates, and you can easily see any old file
changes.

like so:

ls -l ./backup/*/*/*/datafile03.txt ./randomuser/datafile03.txt
-rw-r--r--. 1 randomuser randomuser  150 Jan 21  2016
./backup/201802/20180205/randomuser/datafile03.txt
-rw-r--r--. 1 randomuser randomuser 1019 Dec 19  2017
./backup/201911/20191110/randomuser/datafile03.txt
-rw-r--r--. 1 randomuser randomuser 1104 Sep  5  2019
./backup/202201/20220101/randomuser/datafile03.txt
-rw-r--r--. 1 randomuser randomuser 1874 Dec  7 11:06
./randomuser/datafile03.txt

On Fri, Jan 21, 2022 at 1:02 PM Wols Lists [off-list ref] wrote:

On 18/01/2022 23:00, NeilBrown wrote:

quoted

quoted

Firstly, given that superblocks seem to disappear every now and then,
does anybody have any ideas for something that might help us track it
down? The 1.2 superblock is 4K into the device I believe? So if I copy
the first 8K ( dd if=/dev/sda4 of=sda4.img bs=4K count=2 ) of each
partition, that might help provide any clues as to what's happened to
it? What am I looking for? What is the superblock supposed to look like?

quoted

Yes, 4K offset.  Yes, that dd command will get what you want it to.
It hardly matters what the superblock should looks like, because it
won't be there.  The thing you want to know is: what is there?
i.e.  you see random bytes and need to guess what they mean, so you can
guess where they came from.
Best to post the "od -x" output and crowd-source.

That's exactly what I was thinking. But I was thinking if it had been
damaged rather than destroyed maybe stuff would have been recoverable.

quoted

Are you sure the partition starts haven't changed? Was the array made of
whole-devices or of partitions?

That's what I missed. I forgot my array was on top of dm-integrity, so
although I think of it as sda4, sdb1, sdc4, they each in fact have an
extra layer between them and the raid.

Dunno what or why, but my systemd service that fires that up failed.
status tells me it was killed after 2msec.

So if that wasn't running, the integrity devices weren't there, and
mdadm couldn't start the array.

Oh well, the good thing is that backup drive is on its way. I'm planning
to put plain lvm on it, and write a bunch of services that create backup
volumes then do a overwrite-in-place rsync. So as I keep advising
people, it does an incremental backup, but the COW volumes mean I have
full backups.

Cheers,
Wol