Hello Albrecht,
Sorry for the delay ... your mail got stuck in a Notes "spam filter".
On Friday 22 January 2010 21:11:39 Albrecht Dre=DF wrote:
Are there any "final" conclusions from your tests?
Yes.
=46or the small product using the 2.6 kernel we turned all snooping and
the kernel coherent flag off, which avoided crashes of the FEC and/or
hard disk since we introduced that change.
We are currently investigating problems that we are seeing on the
2.4.25 (DENX and Lite5200B based) boards for a long time. Here we are
having problems with corrupt filesystems and FEC hick ups. In this
case we are using UDMA2, because we cannot yet get MWDMA2 work on
2.4.25, well knowing that there might be a problem with UDMA2 and LPC.
So we also turned of snooping and are currently in the testing phase
(again).
... two compactflash cards with vfat file systems attached to the
ata bus; - a nfs3 network drive, connected via a 100MBit line, on a
Xeon serve
Are you using MWDMA2 with the compact flash cards? What is the load on
the different (DMA) channels? ATA reads or writes?
... a signal processor attached to the localbus, using bestcomm and
the fifo for the bulk transfer
Are you using an own driver, or are you using Grant's SCLPC+SDMA
driver? BD task?
Our latest product uses an SMSC MOST150 Spynic and an FPGA to sample
data from a MOST ring via SCLPC+SDMA (single non BD task from the old
=46reescale Betstcomm API) on the 2.4.25. Here moving from memory
accesses to SCLPC+SDMA helped somewhat, probably by avoiding the
UDMA2/LPC problem by mainly letting the SDMA scheduler do the
scheduling of the LPC traffic, which avoids the LPC arbiter problem
"somehow".
The probability for seeing problems or crashes increases a lot with
the bandwidth. I think, and I might be wrong, esp. when an arbiter or
scheduler (LPC/PCI or SDMA) needs to switch users or tasks. In our
case we have data running with about 3-6 MB/s (avg.) via the LPC to
the hard disks or somewhat more using FTP from the hard disk to the
=46EC.
I did not observe any issues
The filesystem crashes are seldom, but happen often enough to be able
to reproduce them once every 1 or 2 days under heavy load, and to
produce failures in the field, what's even worse. And they
statistically increased a lot wen we ran out of GPIO on the MPC5200B
and then used an CPLD or FPGA to replace them, just a few bits to MUX
SPI lines, but that was enough.
but your statements are making me really nervous...
That was not my intention. The best thing is to run very ugly tests
with very high load for at least 24h. Due to the fact that we see
those problems on different boards we (the SW guys) no longer can
assume self made HW problems (HW guys), esp. when reading Freescale's
advice with the XLB config.
It might happen that we switch to 2.6 on our older products, hoping
that at least the LPC/IDE problem disappears by using MWDMA2 instead
of UDMA2.
Roman
=2D-=20
Roman Fietze Telemotive AG B=FCro M=FChlhausen
Breitwiesen 73347 M=FChlhausen
Tel.: +49(0)7335/18493-45 http://www.telemotive.de
Hi Roman:
Am 03.02.10 07:16 schrieb(en) Roman Fietze:
Sorry for the delay ... your mail got stuck in a Notes "spam filter".
Never mind. I didn't know yet that I'm *such* a nasty guy... ;-)
Are you using MWDMA2 with the compact flash cards? What is the load on the different (DMA) channels? ATA reads or writes?
Actually, I forgot that I have to explicitly enable libata dma on the 5200b, due to the known silicon bugs... I will repeat my tests with the proper configuration, stay tuned.
quoted
... a signal processor attached to the localbus, using bestcomm and the fifo for the bulk transfer
Are you using an own driver, or are you using Grant's SCLPC+SDMA driver? BD task?
Basically Grant's driver, but with a slightly modified variant of the gen_bd task. The signal processor is a LE, and I managed to insert the LE/BE conversion into the bestcomm task (see also <http://patchwork.ozlabs.org/patch/35038/>). Unfortunately, there is no good documentation of the engine; I would like to also shift crc calculation into bestcomm, which seems to be possible in principle, but I never got it running.
The best thing is to run very ugly tests with very high load for at least 24h.
Thanks again for this tip! I hope I manage to run a test over the weekend. Throughput onto the cf cards is not critical for me (so I could live with pio there), but I'm a little afraid I might also see similar effects with fec and the signal processor (in particular, the latter *is* critical).
Thanks, Albrecht.
Hi Roman:
Sorry for the long delay, I had to fix some other stuff first, before I could launch the test... Here is just a short intermediate result.
Am 04.02.10 20:35 schrieb(en) Albrecht Dreß:
Actually, I forgot that I have to explicitly enable libata dma on the 5200b, due to the known silicon bugs... I will repeat my tests with the proper configuration, stay tuned.
quoted
quoted
... a signal processor attached to the localbus, using bestcomm and the fifo for the bulk transfer
Are you using an own driver, or are you using Grant's SCLPC+SDMA driver? BD task?
Basically Grant's driver, but with a slightly modified variant of the gen_bd task. The signal processor is a LE, and I managed to insert the LE/BE conversion into the bestcomm task (see also <http://patchwork.ozlabs.org/patch/35038/>). Unfortunately, there is no good documentation of the engine; I would like to also shift crc calculation into bestcomm, which seems to be possible in principle, but I never got it running.
quoted
The best thing is to run very ugly tests with very high load for at least 24h.
I today launched my test application, on kernel 2.6.32 with a few minor tweaks, which runs 4 threads in parallel, all first writing a number of data blocks, then doing a sync() when appropriate, and reading reading them all back and checking the contents (md5 hash):
- one writes/reads back 256 files of 256k each to a nfs3 share on a Xeon server, using a 100 MBit line;
- one writes/reads back one 1 MByte block using BestComm to a Localbus device (see quote above);
- two write/read back 128 files of 64k each to two CF cards w/ vfat, both attached to the ata (master/slave).
Booting with 'libata.force=mwdma2', this tests reproducibly freezes the system *within a few minutes*, in one case leaving the vfat fs on one card completely broken. The system didn't throw a panic, it was always simply stuck - no response to the serial console, nothing.
Booting *without* this option (i.e. using pio for the cf cards), the system seems to run flawlessly. I will continue the test over the weekend (now active for ~5 hours), but it looks as if I can reproduce your problem. Next week, I'll try your fix (hope I don't wear out the cf cards...), and re-run the test.
Best, Albrecht.