Are you using prism54 or p54 in each kernel?

lsmod | egrep "p54|prism54"

p54pci and p54common are found in both kernel
no modules matching prism54

In fact, a few months ago, after a kernel upgrade, I had to blacklist prism54.
(prism54 is still in the blacklist).

Philippe