abnormal termination of RC and how to deal gracefully with it?
From: Chet Murthy <hidden>
Date: 2011-08-09 05:19:56
I'm trying to figure out how to deal gracefully with abnormal
termination of RC connections. I'm using:
(1) OFED 1.5.2
(2) on Ubuntu Lucid Lynx
(3) and (following a friend's advice) I built my own kernel -without-
any infiniband support, and compiled the OFED kernel modules,
libraries, and userland utilities, myself.
(4) The machines are dual-core opterons
(5) and the HCA is a mellanox adapter
07:00.0 InfiniBand: Mellanox Technologies MT25208 [InfiniHost III Ex] (rev a0)
hca_id: mthca0
fw_ver: 5.2.917
I've been humming along happily, but recently decided to inject some
fault-tolerance into my code. Specifically, I'd like for one end of a
client/server using RC, to be able to crash in arbitrary ways, without
the other end hanging up.
I was told that the way to do this is to use ibv_get_async_event(),
and the documentation appears to confirm this. Basically, I should
spawn a "watchdog thread" that calls this function, and when it
returns, I can then do something (e.g. use a loopback QP to send a
message to the "main thread", to wake it up from the poll or
completion-channel wait that it's stuck in). I prototyped this in the
"rc_pingong.c" sample, though, and had no joy whatsoever. When I fire
up rc_pingopng with these mods, and a large number of iterations, and
then kill the client process, the "watchdog thread" in the server
process stays stuck in ibv_get_async_event(), and doesn't come out,
even after waiting a good number of minutes.
Clearly, I'm not reaping the event on "RC connection timeout"
properly.
I'm attaching the example, in case anybody's interested, but really,
what I'm hoping somebody can point me at, is a worked example --
demonstrating how one end of an RC connection can deal with arbitrary
failures at the other end.
I hope this isn't a newbie sort of question, and would appreciate any
advice anybody could offer.
Thanks in advance,
--chet--
Attachments
- rc_pingpong_ft.c [text/x-csrc] 19991 bytes · preview