Very slow macsec transfers on OpenPOWER hardware

I'm seeing an extreme performance issue with macsec on OpenPOWER (ppc64le) hardware, both with and without encryption.  The base systems are identical POWER9-based hosts with Intel 10Gbe Ethernet adapters; the base link validates at 9.9Gbps but the macsec link maxes out at around 250Mbps without encryption and only 170Mbps with encryption.

Ordinarily I'd suspect a cipher module isn't loaded, but the results without encryption seem to point somewhere else.  In both cases the softirq load is extremely high (100% pegged on one CPU core).

Any debugging tips are appreciated -- I can't imagine a 100-fold performance reduction over the native link is correct. ;)

Thanks!

Quick update:

1.) Disabling encryption actually relieves the load on the softirq handler entirely, but the throughput remains abysmal with extremely high load in the gf128mul_lle function.

2.) Bizarrely, unloading the vmx_crypto (POWER accelerated crypto) module *tripled* the throughput, although 400MB/s is still a long way away from the Gbps range we should be in.

----- Original Message -----

From: "Timothy Pearson" <tpearson@raptorengineering.com>
To: "netdev" <redacted>
Sent: Saturday, August 7, 2021 8:37:17 PM
Subject: Very slow macsec transfers on OpenPOWER hardware

I'm seeing an extreme performance issue with macsec on OpenPOWER (ppc64le)
hardware, both with and without encryption.  The base systems are identical
POWER9-based hosts with Intel 10Gbe Ethernet adapters; the base link validates
at 9.9Gbps but the macsec link maxes out at around 250Mbps without encryption
and only 170Mbps with encryption.

Ordinarily I'd suspect a cipher module isn't loaded, but the results without
encryption seem to point somewhere else.  In both cases the softirq load is
extremely high (100% pegged on one CPU core).

Any debugging tips are appreciated -- I can't imagine a 100-fold performance
reduction over the native link is correct. ;)

Thanks!