Re: [PATCH 0/4] Enable Fault Injection for RTRS

On Apr 6, 2021, at 7:50 AM, Gioh Kim [off-list ref] wrote:

My colleagues and I would like to apply the fault injection
of the Linux to test error handling of RTRS module. RTRS module
consists of client and server modules that are connected via
Infiniband network. So it is important for the client to receive
the error of the server and handle it smoothly.

I am a fan of fault injection. In fact I added a disconnect fault
injector for RPC that's in the kernel now, and it uses debugfs
as its control interface.

But that was years ago. If I were doing this today, I'd consider
kprobes, since fault injection is generally not something that
is consumed by users or administrators in a distributed kernel.

Have you considered injection via kprobes or eBPF instead of
adding permanent code?

When debugfs is enabled, RTRS is able to export interfaces
to fail RTRS client and server.
Following fault injection points are enabled:
- fail a request processing on RTRS client side
- fail a heart-beat transferation on RTRS server side

This patch set is just a starting point. We will enable various
faults and test as many error cases as possible.

Best regards

Gioh Kim (4):
 RDMA/rtrs: Enable the fault-injection
 RDMA/rtrs-clt: Inject a fault at request processing
 RDMA/rtrs-srv: Inject a fault at heart-beat sending
 docs: fault-injection: Add fault-injection manual of RTRS

.../fault-injection/rtrs-fault-injection.rst  | 83 +++++++++++++++++++
drivers/infiniband/ulp/rtrs/Makefile          |  2 +
drivers/infiniband/ulp/rtrs/rtrs-clt-sysfs.c  | 44 ++++++++++
drivers/infiniband/ulp/rtrs/rtrs-clt.c        |  7 ++
drivers/infiniband/ulp/rtrs/rtrs-clt.h        | 13 +++
drivers/infiniband/ulp/rtrs/rtrs-fault.c      | 52 ++++++++++++
drivers/infiniband/ulp/rtrs/rtrs-fault.h      | 28 +++++++
drivers/infiniband/ulp/rtrs/rtrs-srv-sysfs.c  | 44 ++++++++++
drivers/infiniband/ulp/rtrs/rtrs-srv.c        |  5 ++
drivers/infiniband/ulp/rtrs/rtrs-srv.h        | 13 +++
10 files changed, 291 insertions(+)
create mode 100644 Documentation/fault-injection/rtrs-fault-injection.rst
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-fault.c
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-fault.h

-- 
2.25.1

--
Chuck Lever