So both the CPU and the switch are trying to talk to each other but their
frames are not delivered. This still looks like RGMII configuration problem.

Yes, i would say it is.

As it works with my old PHY driver I suspect the problem is at the switches
side somewhere in qca8k_set_pad_ctrl or qca8k_setup in the qca8k driver.

Any other ideas?

I would probably add code to dump all the qca8k registers. Compare the
values for your working setup and your non-working setup. Hopefully
they are not too different, and you can quickly get to the bits which
matter.

     Andrew