IB/iSER major problems with Linux 3.0 and Solaris targets

Hi list, hi Or,

at the moment we use our kernel 3.0 ported OFA kernel modules from
OFED-1.5.4 to make iSER work together with our kernel 3.0 ported
open-iscsi 2.0.869 and Solaris 11 COMSTAR targets.

Now, we've got the problem that ib_iser from in there ignores its
"debug_level" parameter. /sys/module/ib_iser/parameters/debug_level
shows "0" but it spams the whole kernel log with debug messages. Seems
to be a code bug.

We've also tested a 3.0.15 mainline kernel with in-tree IB modules
together with the OFED-1.5.4 user-space and this has much better IPoIB
performance than the kernel stuff from OFED. So, we want to use them
instead, but there is the same problem with the iSER debug messages and
iSER doesn't work together with our Solaris 11 COMSTAR targets.

We've tested this with open-iscsi (2.0.872, commit
4323e342d2c9fb8ed7233ce855001c189ec55b23) user-space.
TCP is O.K. but iSER reports an error while login attempt:
"iscsiadm: initiator reported error (11 - iSCSI PDU timed out)"

The sent PDUs from iscsid debugging are the same, but there is an IO
page fault in the kernel log. I've attached the relevant part and the
iscsid log.

This looks interesting:
"iser: iser_drain_tx_cq:tx id ffff88402391f898 status 4 vend_err 57"

Or, could you please investigate/explain?

It is a pain that we need both: working iSER and IPoIB traffic with good
performance.

Cheers,

Sebastian


On 19/12/11 10:14, Sebastian Riemer wrote:

Hi list,

I've already sent this to the open-iscsi mailing list, but I guess
this is more relevant for linux-rdma.

Finally I've got IB/iSER running on Debian Squeeze with Linux kernel 3.0
smoothly.

The problem was that we did not have the suitable OFED for our kernel
and we did not use the open-iscsi from OFED. Kernel 3.0 is supported
since OFED-1.5.4 from 2011-12-05.

So, I've taken the 1.5.2-based stuff from Debian/Experimental and I've
updated it to 1.5.4 from OFA. Then, I've noticed that Debian doesn't
build "ib_iser" in the OFA kernel source and that they don't build the
open-iscsi kernel/user-space code - I made it do so.

The next problem was that open-iscsi kernel code in OFED-1.5.4 is for <=
2.6.32 based RedHat distributions. I had to port the source from 2.6.30
to 3.0 due to kernel API changes. OFA even forgot libiscsi_tcp.[ch] in
OFED-1.5.4. So, I had to import it from 2.6.30 mainline.
I did so, because we wanted to compare TCP and iSER speed over
InfiniBand. Our Solaris COMSTAR targets provide both.

After fixing the kernel, there was still a problem in the open-iscsi
2.0.869 user-space from OFED. Some sysfs magic has changed - so that the
iSCSI host number couldn't be found.

After fixing that, it worked for me.

Cheers,

Sebastian