Linux hanging on Xilinx SystemACE

Hey,

Using the powerpc development tree of Linux 2.4, I am trying to boot my
system from CompactFlash using Xilinx SystemACE. My compact flash card
has two partitions, a 16MB FAT16 that holds the combination FPGA image /
Linux Kernel ELF file, and an Ext2 partition that holds the root file
system. The system starts the boot process, uncompresses the Linux
kernel and begins loading drivers. Part way into this process, it
conducts a partition check of the drive being reported to it by
SystemACE, however, it hangs at that point. No kernel panic, no error
message, it simply hangs. Here is the output at that point...

Partition check:
 xsysacea:
 
what I am trying to find out is if this problem has been seen/fixed in
the past? or did I format the CF card incorrectly?
 
Clinton Thomas
cthomas@soneticom.com
 

On 8/16/06, Clint Thomas [off-list ref] wrote:

Hey,

Using the powerpc development tree of Linux 2.4, I am trying to boot my
system from CompactFlash using Xilinx SystemACE. My compact flash card has
two partitions, a 16MB FAT16 that holds the combination FPGA image / Linux
Kernel ELF file, and an Ext2 partition that holds the root file system. The
system starts the boot process, uncompresses the Linux kernel and begins
loading drivers. Part way into this process, it conducts a partition check
of the drive being reported to it by SystemACE, however, it hangs at that
point. No kernel panic, no error message, it simply hangs. Here is the
output at that point...

Partition check:
 xsysacea:

what I am trying to find out is if this problem has been seen/fixed in the
past? or did I format the CF card incorrectly?

Checking partitions is a user-space activity (fsck).  Remove it from
your init scripts.  Besides, unless your using a microdrive, your ext2
rootfs should be mounted read-only which greatly reduces the need for
fsck.  (because FLASH will wear out after too many writes)

Cheers,
g.
-- 
Grant Likely, B.Sc. P.Eng.
Secret Lab Technologies Ltd.
grant.likely@secretlab.ca
(403) 399-0195

Clint,

Using the powerpc development tree of Linux 2.4, I am trying to boot my 

system from CompactFlash using Xilinx SystemACE. My compact flash card has 
two partitions, a 16MB FAT16 that holds the combination FPGA image / Linux 
Kernel ELF file, and an Ext2 partition that holds the root file system. 
The system starts the boot process, uncompresses the Linux kernel and 
begins loading drivers. Part way into this process, it conducts a 
partition check of the drive being reported to it by SystemACE, however, 
it hangs at that point. No kernel panic, no error message, it simply 
hangs. Here is the output at that point...

Partition check:
 xsysacea:

what I am trying to find out is if this problem has been seen/fixed in 

the past? or did I format the CF card incorrectly?

I have a system that uses the SystemAce in a similar way.  I was also 
having lockups.  After a lot of digging, I found a patch by John Masters 
(e-mail unknown) to the MontaVista SystemAce driver in the 2.4 kernel that 
disables the use of interrupts and runs the SystemAce in a polled mode. 
Performance is not that great, but at least it does not crash with the 
patch.  As I recall, the reason that the unpatched driver crashed is that 
the SystemAce is issuing more than one interrupt upon completion of a 
sector read or write.  Apparently, the Xilinx ML300 board works fine 
without this patch while other eval boards like the Memec DS-BD-2VPxx 
crash without the patch.

This still may not be your problem, but the patch may help later on.

I've attached the patched file.  The file is from 
./drivers/block/xilinx_sysace/adapter.c in the 2.4.30 from MontaVista.

Keith

On Wed, 2006-08-16 at 17:06 -0400, Clint Thomas wrote:

Using the powerpc development tree of Linux 2.4, I am trying to boot
my system from CompactFlash using Xilinx SystemACE. My compact flash
card has two partitions, a 16MB FAT16 that holds the combination FPGA
image / Linux Kernel ELF file, and an Ext2 partition that holds the
root file system. The system starts the boot process, uncompresses the
Linux kernel and begins loading drivers. Part way into this process,
it conducts a partition check of the drive being reported to it by
SystemACE, however, it hangs at that point. No kernel panic, no error
message, it simply hangs. Here is the output at that point...

Partition check:
 xsysacea:
 
what I am trying to find out is if this problem has been seen/fixed in
the past? or did I format the CF card incorrectly?

Can u-boot see the partition formatted with ext2? 

Can you mount an NFS root filesystem and access the card normally?  Or
is it just when you use it as your rootfs?

How did you create you partitions and format the CF card?

And like somebody else mentioned, if you are really going to use this
for an embedded system, you are going to want to rethink your
partitioning scheme.

Maybe something like:

p1 fat12 - kernel and binary image
p2 ext2 - read only rootfs
p3 ext3 - non volatile, slow rate data rootfs
p4 tmpfs - volatile, high rate data rootfs


Jeff Angielski
The PTR Group

On 8/16/06, Jeff Angielski [off-list ref] wrote:

And like somebody else mentioned, if you are really going to use this
for an embedded system, you are going to want to rethink your
partitioning scheme.

Maybe something like:

p1 fat12 - kernel and binary image
p2 ext2 - read only rootfs
p3 ext3 - non volatile, slow rate data rootfs
p4 tmpfs - volatile, high rate data rootfs

Or, if you have enough ram (and a small enough rootfs footprint); put
the rootfs into an initramfs and leave the CF alone entirely after
boot.  On small systems, I store all config parameters in a flat file
on the fat partition, and only write it out when it needs to save a
new configuration.  That way the entire system consists of three
files; a kernel, a rootfs image and a config file.  Makes managing
updates very easy.  :)

g.

-- 
Grant Likely, B.Sc. P.Eng.
Secret Lab Technologies Ltd.
grant.likely@secretlab.ca
(403) 399-0195

On Wed, 2006-08-16 at 15:48 -0600, Grant Likely wrote:

On 8/16/06, Clint Thomas [off-list ref] wrote:

quoted

Hey,

Using the powerpc development tree of Linux 2.4, I am trying to boot my
system from CompactFlash using Xilinx SystemACE. My compact flash card has
two partitions, a 16MB FAT16 that holds the combination FPGA image / Linux
Kernel ELF file, and an Ext2 partition that holds the root file system. The
system starts the boot process, uncompresses the Linux kernel and begins
loading drivers. Part way into this process, it conducts a partition check
of the drive being reported to it by SystemACE, however, it hangs at that
point. No kernel panic, no error message, it simply hangs. Here is the
output at that point...

Partition check:
 xsysacea:

what I am trying to find out is if this problem has been seen/fixed in the
past? or did I format the CF card incorrectly?

I forgot to mention that we used to see this problem when the identify
command that is sent during intialization fails.  The driver is written
in such a way that if any of this fails, the system hangs because it
sits in a polling loop waiting for the correct response.  There are no
timeout failures... :(  In our case we saw this error because we forgot
to put a CF into the system [usually during development with NFS
rootfs].

It is fairly easy to printk() the drivers init code to find out which
step is stuck in the polling loop.

Checking partitions is a user-space activity (fsck).  Remove it from
your init scripts.  Besides, unless your using a microdrive, your ext2
rootfs should be mounted read-only which greatly reduces the need for
fsck.  (because FLASH will wear out after too many writes)

The partition check he is referring to is part of a block device driver
initialization.  It is not fsck.  If he were only so luck to be that far
in the startup sequence... :)



Jeff Angielski
The PTR Group

On 8/16/06, Jeff Angielski [off-list ref] wrote:

On Wed, 2006-08-16 at 15:48 -0600, Grant Likely wrote:

quoted

Checking partitions is a user-space activity (fsck).  Remove it from
your init scripts.  Besides, unless your using a microdrive, your ext2
rootfs should be mounted read-only which greatly reduces the need for
fsck.  (because FLASH will wear out after too many writes)

The partition check he is referring to is part of a block device driver
initialization.  It is not fsck.  If he were only so luck to be that far
in the startup sequence... :)

heh; I grepped the 2.6 kernel tree for "Partition check" and didn't
find anything, and I assumed that it wasn't in 2.4 either.  Therefore
it must be a fsck message...

You know what they say about assuming.

oops

g.

-- 
Grant Likely, B.Sc. P.Eng.
Secret Lab Technologies Ltd.
grant.likely@secretlab.ca
(403) 399-0195

The adapter.c file fixed the problem, however running in polling I/O
mode is far too slow. Since this is not an option for successful
operation of our machine, do you know of a driver fix for this problem?
Perhaps a newer version of the linux source tree is necessary. I'm using
the Linuxppc_2_4_devel tree available from bkbits/mvista. Thanks for the
help so far!

Clint Thomas

-----Original Message-----
From: linuxppc-embedded-bounces+cthomas=3Dsoneticom.com@ozlabs.org
[mailto:linuxppc-embedded-bounces+cthomas=3Dsoneticom.com@ozlabs.org] On
Behalf Of Keith J Outwater
Sent: Wednesday, August 16, 2006 6:02 PM
To: linuxppc-embedded@ozlabs.org
Subject: Re: Linux hanging on Xilinx SystemACE

Clint,

Using the powerpc development tree of Linux 2.4, I am trying to boot=20
my

system from CompactFlash using Xilinx SystemACE. My compact flash card
has two partitions, a 16MB FAT16 that holds the combination FPGA image /
Linux Kernel ELF file, and an Ext2 partition that holds the root file
system.=20
The system starts the boot process, uncompresses the Linux kernel and
begins loading drivers. Part way into this process, it conducts a
partition check of the drive being reported to it by SystemACE, however,
it hangs at that point. No kernel panic, no error message, it simply
hangs. Here is the output at that point...

=20
Partition check:
 xsysacea:
=20
what I am trying to find out is if this problem has been seen/fixed in

the past? or did I format the CF card incorrectly?

=20

I have a system that uses the SystemAce in a similar way.  I was also
having lockups.  After a lot of digging, I found a patch by John Masters
(e-mail unknown) to the MontaVista SystemAce driver in the 2.4 kernel
that disables the use of interrupts and runs the SystemAce in a polled
mode.=20
Performance is not that great, but at least it does not crash with the
patch.  As I recall, the reason that the unpatched driver crashed is
that the SystemAce is issuing more than one interrupt upon completion of
a sector read or write.  Apparently, the Xilinx ML300 board works fine
without this patch while other eval boards like the Memec DS-BD-2VPxx
crash without the patch.

This still may not be your problem, but the patch may help later on.

I've attached the patched file.  The file is from
./drivers/block/xilinx_sysace/adapter.c in the 2.4.30 from MontaVista.

Keith

The adapter.c file fixed the problem, however running in polling I/O
mode is far too slow. Since this is not an option for successful
operation of our machine, do you know of a driver fix for this problem?
Perhaps a newer version of the linux source tree is necessary. I'm using
the Linuxppc_2_4_devel tree available from bkbits/mvista. Thanks for the
help so far!

Sorry, I am not aware of any driver fix to make interrupt mode work.
Unfortunately, the use of the SystemACE under Linux for access to CF
file systems appears to be a niche application, so it does not receive
a lot of attention.

I recall someone on the net stating that they had observed the SystemAce
issuing more than one interrupt in response to sector transfers.  The
SystemACE is a strange beast (as you may have already observed!) and
perhaps it is being configured in hardware or software differently in your
design versus the Xilinx ML300 design.
For my particular application, I used the CF card to hold a root
filesystem image that was placed in RAM on boot, so I did not see a big
hit during operation.  Basically, the mediocre performance of the driver
in polled mode was good enough for me.  Maybe some day I'll return to that
problem, but right now other issues are more pressing.  As I stated
originally, the SystemACE driver appears to work fine in interrupt
mode for ML300 boards, but dies with Memec eval boards.  Perhaps you can
download the relevant designs for each board and compare them.  If you
do this, be sure to check the schematics as well as the configuration
of the SystemACE and the interrupt controller IP blocks in the Xilinx EDK
tool.  Maybe this will uncover something obvious.  Also, if MontaVista
supports their kernel on the Memec board, they may have seen this problem
and maybe even have fixed it.  I also used the mvista kernel and there is
support for a Memec board in there, although I don't know the exact part 
number.

Keith

Clint Thomas

-----Original Message-----
From: linuxppc-embedded-bounces+cthomas=soneticom.com@ozlabs.org
[mailto:linuxppc-embedded-bounces+cthomas=soneticom.com@ozlabs.org] On
Behalf Of Keith J Outwater
Sent: Wednesday, August 16, 2006 6:02 PM
To: linuxppc-embedded@ozlabs.org
Subject: Re: Linux hanging on Xilinx SystemACE

Clint,

quoted

Using the powerpc development tree of Linux 2.4, I am trying to boot 
my

system from CompactFlash using Xilinx SystemACE. My compact flash card
has two partitions, a 16MB FAT16 that holds the combination FPGA image /
Linux Kernel ELF file, and an Ext2 partition that holds the root file
system. 
The system starts the boot process, uncompresses the Linux kernel and
begins loading drivers. Part way into this process, it conducts a
partition check of the drive being reported to it by SystemACE, however,
it hangs at that point. No kernel panic, no error message, it simply
hangs. Here is the output at that point...

quoted

Partition check:
 xsysacea:

what I am trying to find out is if this problem has been seen/fixed in

the past? or did I format the CF card incorrectly?

quoted

I have a system that uses the SystemAce in a similar way.  I was also
having lockups.  After a lot of digging, I found a patch by John Masters
(e-mail unknown) to the MontaVista SystemAce driver in the 2.4 kernel
that disables the use of interrupts and runs the SystemAce in a polled
mode. 
Performance is not that great, but at least it does not crash with the
patch.  As I recall, the reason that the unpatched driver crashed is
that the SystemAce is issuing more than one interrupt upon completion of
a sector read or write.  Apparently, the Xilinx ML300 board works fine
without this patch while other eval boards like the Memec DS-BD-2VPxx
crash without the patch.

This still may not be your problem, but the patch may help later on.

I've attached the patched file.  The file is from
./drivers/block/xilinx_sysace/adapter.c in the 2.4.30 from MontaVista.

Keith