[...]
Now the interesting (and new for me) part is: if I remove the at803x driver from
the system and use the generic phy instead, the problem does not happen (or at
least not while running for one full day). [...]
Update: the failure also happened with genphy after running for another day.