Re: 2.6.33.6-rt28 kernel oops while stressing network

Hi John,

First of all, you will want to become familiar with the 'chrt' command
(available also as an applet part of busybox).  That command (plus some
of the finer details in optional output from the 'ps' command) is how
you will tweak and experiment on a target system.

Then the attached may be of use (I attached them because they are
small).  rtctl is a package that originates from Red Hat.  If you are
using busybox shell (rather than full blown bash), then apply the
attached patch to it.

Extract the rpm something like...

rpm2cpio rtctl-1.7-1.el5rt.src.rpm | cpio --extract --make-directories

...assuming you have rpm2cpio installed on your host.  Then extract any
tarballs that were bundled inside it.  Then apply the attached patch to
it (if not using bash on your target).

Then install to your target rootfs staging area something like...

install -D -m 755 rtctl $MY_STAGING_DIR_PATH/usr/sbin/rtctl
install -D -m 644 rtgroups $MY_STAGING_DIR_PATH/etc/rtgroups
install -D -m 755 rtctl.sysconfig
$MY_STAGING_DIR_PATH/etc/sysconfig/rtctl 

The package includes more but that's all I've ported and tried out so
far on a uclibc/buildroot style embedded Linux with RT kernel so far.

Edit the rtgroups file accordingly to implement an RT policy that works
for you (based on outcome of your experiments using chrt).

What hasn't been implemented yet is how to hook rtctl into your system
initialization scripts.  The stock content from the Red Hat package is
for their real time distro.  You'd need to invoke it from your scripts
accordingly.


Regards,

Darcy

On Fri, 2010-08-13 at 12:16 -0700, John Culvertson wrote:

Thanks for the pointers.  I have seen others mention adjusting the
soft irq thread priorities, etc.  Can you shed any light on how to go
about doing that?  Is that in the kernel configuration, or do you have
to modify the kernel source?

On Fri, Aug 13, 2010 at 2:14 PM, Darcy Watkins [off-list ref]
wrote:

quoted

Hi John,

In the 'make menuconfig', look for and double check any config

settings related to IRQ sharing, PCI, etc.  There are a lot of
tweaks.  Probably enough to write a PhD dissertation about.

quoted

The other thing, if it only happens under high network stress, you

may want to check into tweaking the real time priorities of your
kernel threads, user space program threads and even IRQ threads.  RT
kernels tend to treat priorities more strictly so it is possible for a
high priority thread to hog the CPU and deplete resources before a
lower priority thread processes them and frees up the resources.

quoted

Regards,

Darcy


-----Original Message-----
From: John Culvertson [mailto:jculvertson@gmail.com]
Sent: Friday, August 13, 2010 11:06 AM
To: Darcy Watkins
Subject: Re: 2.6.33.6-rt28 kernel oops while stressing network

Thanks for the suggestions.  I have tried the unpatched 2.6.33.7
kernel, and the problem does not occur.  The hardware is a single
board industrial computer with the network controllers onboard, so I
cannot easily try different NICs.  I have not seen the problem occur
with only one port in use, but I have not tested that long enough to
be positive.

One thing that may be a little odd about this computer is that both
Ethernet controllers (Intel 82559) share the same PCI interrupt.
Interrupt sharing should be OK, but since adjacent PCI slots in

normal

quoted

PCs generally use different interrupts, it may not occur often in
other systems.

On Fri, Aug 13, 2010 at 1:56 PM, Darcy Watkins

[off-list ref] wrote:

quoted

quoted

Hi John,

I use Fedora 13 which has 2.6.33.6 as the kernel (without RT).

My machine has three net i/f in it.  Two PCI net cards with Realtek
chipset and Intel PRO built into the mainboard's chipset.

When I installed Fedora using the netboot USB flash drive, it

insisted

quoted

quoted

on using one of the Realtek interfaces for Internet connection so

that

quoted

quoted

is my eth0.  All fine.

More recently, I activated the other two net i/f for private LAN to
target HW test network.  I set one to 10.0.0.1 and the other to
192.168.101.4 and connected them to the target network.  Note that

they

quoted

quoted

were both connected to the same network switch.

Shortly after that, the system froze.  After reboot it would run

for a

quoted

quoted

while and then freeze.  I unplugged the net i/f based on the Intel

PRO

quoted

quoted

and all has been fine since.

I mention all this because you once mentioned you were using two

Intel

quoted

quoted

net i/f.  It may not even be RT related.

I suggest you try (not in any particular order, but each on its

own)...

quoted

quoted

  - running with only one net i/f connected
  - building a side-by-side vanilla kernel 2.6.33.7 (without RT

patch)

quoted

quoted

and running your two net i/f without RT
  - using different net i/f cards (say based on Realtek or

something

quoted

quoted

other than Intel) try running it with RT

If you see your system behavior change related to any of these, it
possibly may not be RT patch related (or it could be tied to a

specific

quoted

quoted

driver).

Regards,

Darcy

-----Original Message-----
From: linux-rt-users-owner@vger.kernel.org
[mailto:linux-rt-users-owner@vger.kernel.org] On Behalf Of John
Culvertson
Sent: Friday, August 13, 2010 10:38 AM
To: linux-rt-users@vger.kernel.org
Subject: Re: 2.6.33.6-rt28 kernel oops while stressing network

Since it was my understanding that x86 was the most mature and

stable

quoted

quoted

architecture for preempt-rt, I was surprised when I immediately
encountered problems.  Is this typical when trying the patches on a
new platform?  Like I mentioned before, I am a newbie with

preempt-rt.

quoted

quoted

On Wed, Aug 11, 2010 at 12:53 PM, John Culvertson
[off-list ref] wrote:

quoted

I updated to 2.6.33.7-rt29, and I am seeing similar symptoms.

[ 2120.781166] BUG: unable to handle kernel paging request at

c11cd497

quoted

quoted

quoted

[ 2120.784018] IP: [<c11d5ce2>] tcp_set_skb_tso_segs+0x33/0x85
[ 2120.784018] *pde = 1d7f6063 *pte = 011cd161
[ 2120.784018] Oops: 0003 [#1] PREEMPT

--
To unsubscribe from this list: send the line "unsubscribe
linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html