quoted

FWIW, in my experience the hardware independent parts of the 
networking stack are
very stable and the problem is almost always with the drivers, or with 
the IP
configuration (e.g. two interfaces on the same subnet).

    I have no doubt it is with the driver.  I am somewhat fortunate in 
this instance that I have a nearly identical setup - this is an FPGA 
based system
    I can swap the FPGA firmware, get an almost identical kernel with a 
slightly different NIC, and everything works - same cables, same IP's,
    Same switch, The only things different are the NIC and its driver. 
Even the Linux kernels are identical - except the NIC driver.
   
    BUT so is the data received and passed on to the kernel (outside 
random differences in the padding of the ARP packet)
    One works the other doesn't.

Well ethernet device drivers contain multiple arp supporting methods,
e.g. header_cache, header_cache_update, hard_header_parse, etc etc.
Generally driver writers don't need to concern themselves about these
as they are assigned to generic handlers by ether_setup().  However,
your problematic driver may do something different.

Given this problem appears to be driver specific rather than PPC
specific your best bet is to try and contact the author.  BTW, I don't
think you've said which driver you are using, a key piece of info....

Alex