Strange Badness with RT Spin Lock

Hello,

I have a very unusual bug I have been trying to get to the bottom of.

First the background...

   AMCC PPC405EP on embedded network device (simpler than a Taihu)
   Kernel 2.6.25.8-rt7 with incremental patches up to 2.6.25.13
   mPCI slot with a WiMax radio card

This is an RT-Preemption kernel.

In my system, in a real time radio (WiMax) driver, I have code that
makes use of a spinlock to control access to a tree structure...

spinlock_t tree_lock;

...

Then I added the following to check if my code is about to generate a
kernel oops that normally happens when a spinlock is locked twice by the
same task.  I did this because I can't find any logical code path that
would cause it to be locked twice so now I look for strange badness.

Note that in an RT Preemption kernel, the spinlock is implemented using
rtmutex.

#define USE_MUTEX_CHECK_HACK	1
#ifdef USE_MUTEX_CHECK_HACK
// This is just a temporary debug hack lifted from rtmutex code in the kernel.
#define RT_MUTEX_OWNER_MASKALL	3UL
static inline struct task_struct *rt_mutex_owner(spinlock_t * lock)
{
	struct rt_mutex * rtlock = (struct rt_mutex *)lock;
	return (struct task_struct *)
		((unsigned long)rtlock->owner & ~RT_MUTEX_OWNER_MASKALL);
}

#define MUTEX_CHECK_SAVE_LOCKED \
   spinlock_t saved_locked = tree_lock;

192 more lines

Further to what I reported earlier regarding my unusual bug, adding a
memory barrier such as shown below also prevents the oops (and is a lot
cleaner than putting in unused dummy static instance(s) of spinlocks).

This has me scratching my head.  Isn't the PPC405EP a uni-processor?  I
suppose my problem could be an interaction of RT-Preemption with
predictive loads / deferred stores or perhaps there is a memory barrier
related bug in the spinlock/rtmutex code?.

        // We dispatch the PDU.
         ...

#ifdef USE_MUTEX_CHECK_HACK
         MUTEX_CHECK_SAVE_LOCKED;
#endif
         spin_unlock(&tree_lock);
#ifdef USE_MUTEX_CHECK_HACK
         rt_mutex_owner_check("process_packed_pdu() - no frag - before
cblk()");
         MUTEX_CHECK_SAVE_UNLOCKED;
#endif
         clbk(new_skb, gen_header_p->cid);

         barrier(); // memory barrier

#ifdef USE_MUTEX_CHECK_HACK
         MUTEX_CHECK_DUMP_ON_BUG;
         rt_mutex_owner_check("process_packed_pdu() - no frag - after
cblk()");
#endif
         spin_lock(&tree_lock);

On Fri, 2008-08-08 at 12:56 -0700, Darcy Watkins wrote:

Hello,

I have a very unusual bug I have been trying to get to the bottom of.

--snip!--


-- 


Regards,

Darcy

--------------
Darcy L. Watkins - Senior Software Developer
Tranzeo Wireless Technologies, Inc.
19273 Fraser Way, Pitt Meadows, BC, Canada V3Y 2V4
T:604-460-6002 ext:410
http://www.tranzeo.com

I have an update...

First please disregard the follow-up email I sent mentioning memory
barriers.  That blew up shortly after I sent out the email.

The problem appears to be resolved when I declare the spinlock variable
using the __cacheline_aligned attribute (that is without the _in_smp
suffix).

Does anyone know of an explanation as to why a spinlock may need to be
cacheline aligned in a uniprocessor system?


Regards,

Darcy


On Fri, 2008-08-08 at 12:56 -0700, Darcy Watkins wrote:

Hello,

I have a very unusual bug I have been trying to get to the bottom of.

First the background...

   AMCC PPC405EP on embedded network device (simpler than a Taihu)
   Kernel 2.6.25.8-rt7 with incremental patches up to 2.6.25.13
   mPCI slot with a WiMax radio card

This is an RT-Preemption kernel.

In my system, in a real time radio (WiMax) driver, I have code that
makes use of a spinlock to control access to a tree structure...

spinlock_t tree_lock;

-- 


Regards,

Darcy

--------------
Darcy L. Watkins - Senior Software Developer
Tranzeo Wireless Technologies, Inc.
19273 Fraser Way, Pitt Meadows, BC, Canada V3Y 2V4
T:604-460-6002 ext:410
http://www.tranzeo.com