From: Eric Leblond <hidden> Date: 2017-12-15 10:24:51
Hello,
When using an ixgbe card with Suricata we are using the following
commands to get a symmetric hash on RSS load balancing:
./set_irq_affinity 0-15 eth3
ethtool -X eth3 hkey 6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A equal 16
ethtool -x eth3
ethtool -n eth3
Then we start Suricata.
In my current experiment on XDP, I have Suricata that inject the eBPF
program when starting. The consequence of that when using an ixgbe card
is that the load balancing get reset and all interrupts are reaching
the first core.
My analysis is that in the ixgbe_xdp_setup() function we call
ixgbe_setup_tc() that reset the hash tuning parameter.
If we run the affinity script and the ethtool commands after XDP is
loaded then things are going normal again. But this is not an optimal
behavior.
Is this really what is happening ? Is there a known workaround for this
issue ?
BR,
--
Eric Leblond [off-list ref]
From: David Miller <davem@davemloft.net> Date: 2017-12-15 15:53:39
From: Eric Leblond <redacted>
Date: Fri, 15 Dec 2017 11:24:46 +0100
Hello,
When using an ixgbe card with Suricata we are using the following
commands to get a symmetric hash on RSS load balancing:
./set_irq_affinity 0-15 eth3
ethtool -X eth3 hkey 6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A equal 16
ethtool -x eth3
ethtool -n eth3
Then we start Suricata.
In my current experiment on XDP, I have Suricata that inject the eBPF
program when starting. The consequence of that when using an ixgbe card
is that the load balancing get reset and all interrupts are reaching
the first core.
This definitely should _not_ be a side effect of enabling XDP on a device.
From: John Fastabend <john.fastabend@gmail.com> Date: 2017-12-15 16:03:18
On 12/15/2017 07:53 AM, David Miller wrote:
From: Eric Leblond <redacted>
Date: Fri, 15 Dec 2017 11:24:46 +0100
quoted
Hello,
When using an ixgbe card with Suricata we are using the following
commands to get a symmetric hash on RSS load balancing:
./set_irq_affinity 0-15 eth3
ethtool -X eth3 hkey 6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A equal 16
ethtool -x eth3
ethtool -n eth3
Then we start Suricata.
In my current experiment on XDP, I have Suricata that inject the eBPF
program when starting. The consequence of that when using an ixgbe card
is that the load balancing get reset and all interrupts are reaching
the first core.
This definitely should _not_ be a side effect of enabling XDP on a device.
Agreed, CC Emil and Alex we should restore these settings after the
reconfiguration done to support a queue per core.
.John
From: Alexander Duyck <hidden> Date: 2017-12-15 16:51:13
On Fri, Dec 15, 2017 at 8:03 AM, John Fastabend
[off-list ref] wrote:
On 12/15/2017 07:53 AM, David Miller wrote:
quoted
From: Eric Leblond <redacted>
Date: Fri, 15 Dec 2017 11:24:46 +0100
quoted
Hello,
When using an ixgbe card with Suricata we are using the following
commands to get a symmetric hash on RSS load balancing:
./set_irq_affinity 0-15 eth3
ethtool -X eth3 hkey 6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A equal 16
ethtool -x eth3
ethtool -n eth3
Then we start Suricata.
In my current experiment on XDP, I have Suricata that inject the eBPF
program when starting. The consequence of that when using an ixgbe card
is that the load balancing get reset and all interrupts are reaching
the first core.
This definitely should _not_ be a side effect of enabling XDP on a device.
Agreed, CC Emil and Alex we should restore these settings after the
reconfiguration done to support a queue per core.
.John
So the interrupt configuration has to get reset since we have to
assign 2 Tx queues for every Rx queue instead of the 1-1 that was
previously there. That is a natural consequence of rearranging the
queues as currently happens. The issue is the q_vectors themselves
have to be reallocated. The only way to not make that happen would be
to pre-allocate the Tx queues for XDP always.
Also just to be clear we are talking about the interrupts being reset,
not the RSS key right? I just want to make sure that is what we are
talking about.
Thanks.
- Alex
From: Peter Manev <hidden> Date: 2017-12-15 16:56:31
On 15 Dec 2017, at 17:51, Alexander Duyck [off-list ref] wrote:
On Fri, Dec 15, 2017 at 8:03 AM, John Fastabend
[off-list ref] wrote:
quoted
On 12/15/2017 07:53 AM, David Miller wrote:
quoted
From: Eric Leblond <redacted>
Date: Fri, 15 Dec 2017 11:24:46 +0100
quoted
Hello,
When using an ixgbe card with Suricata we are using the following
commands to get a symmetric hash on RSS load balancing:
./set_irq_affinity 0-15 eth3
ethtool -X eth3 hkey 6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A equal 16
ethtool -x eth3
ethtool -n eth3
Then we start Suricata.
In my current experiment on XDP, I have Suricata that inject the eBPF
program when starting. The consequence of that when using an ixgbe card
is that the load balancing get reset and all interrupts are reaching
the first core.
This definitely should _not_ be a side effect of enabling XDP on a device.
Agreed, CC Emil and Alex we should restore these settings after the
reconfiguration done to support a queue per core.
.John
So the interrupt configuration has to get reset since we have to
assign 2 Tx queues for every Rx queue instead of the 1-1 that was
previously there. That is a natural consequence of rearranging the
queues as currently happens. The issue is the q_vectors themselves
have to be reallocated. The only way to not make that happen would be
to pre-allocate the Tx queues for XDP always.
Also just to be clear we are talking about the interrupts being reset,
not the RSS key right? I just want to make sure that is what we are
talking about.
Yes.
From the tests we did I only observed the IRQs being all reset to the first CPU after Suricata started.
From: Peter Manev <hidden> Date: 2018-01-25 13:09:47
On Fri, Dec 15, 2017 at 5:56 PM, Peter Manev [off-list ref] wrote:
quoted
On 15 Dec 2017, at 17:51, Alexander Duyck [off-list ref] wrote:
On Fri, Dec 15, 2017 at 8:03 AM, John Fastabend
[off-list ref] wrote:
quoted
On 12/15/2017 07:53 AM, David Miller wrote:
quoted
From: Eric Leblond <redacted>
Date: Fri, 15 Dec 2017 11:24:46 +0100
quoted
Hello,
When using an ixgbe card with Suricata we are using the following
commands to get a symmetric hash on RSS load balancing:
./set_irq_affinity 0-15 eth3
ethtool -X eth3 hkey 6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A equal 16
ethtool -x eth3
ethtool -n eth3
Then we start Suricata.
In my current experiment on XDP, I have Suricata that inject the eBPF
program when starting. The consequence of that when using an ixgbe card
is that the load balancing get reset and all interrupts are reaching
the first core.
This definitely should _not_ be a side effect of enabling XDP on a device.
Agreed, CC Emil and Alex we should restore these settings after the
reconfiguration done to support a queue per core.
.John
So the interrupt configuration has to get reset since we have to
assign 2 Tx queues for every Rx queue instead of the 1-1 that was
previously there. That is a natural consequence of rearranging the
queues as currently happens. The issue is the q_vectors themselves
have to be reallocated. The only way to not make that happen would be
to pre-allocate the Tx queues for XDP always.
Also just to be clear we are talking about the interrupts being reset,
not the RSS key right? I just want to make sure that is what we are
talking about.
Yes.
From the tests we did I only observed the IRQs being all reset to the first CPU after Suricata started.
quoted
Thanks.
- Alex
Hi,
We were wondering if there is any follow up/potential solution for that?
If there is something we could help out testing with regards to that
- please let us know.
Thank you
--
Regards,
Peter Manev
From: Alexander Duyck <hidden> Date: 2018-01-25 15:07:30
On Thu, Jan 25, 2018 at 5:09 AM, Peter Manev [off-list ref] wrote:
On Fri, Dec 15, 2017 at 5:56 PM, Peter Manev [off-list ref] wrote:
quoted
quoted
On 15 Dec 2017, at 17:51, Alexander Duyck [off-list ref] wrote:
On Fri, Dec 15, 2017 at 8:03 AM, John Fastabend
[off-list ref] wrote:
quoted
On 12/15/2017 07:53 AM, David Miller wrote:
quoted
From: Eric Leblond <redacted>
Date: Fri, 15 Dec 2017 11:24:46 +0100
quoted
Hello,
When using an ixgbe card with Suricata we are using the following
commands to get a symmetric hash on RSS load balancing:
./set_irq_affinity 0-15 eth3
ethtool -X eth3 hkey 6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A equal 16
ethtool -x eth3
ethtool -n eth3
Then we start Suricata.
In my current experiment on XDP, I have Suricata that inject the eBPF
program when starting. The consequence of that when using an ixgbe card
is that the load balancing get reset and all interrupts are reaching
the first core.
This definitely should _not_ be a side effect of enabling XDP on a device.
Agreed, CC Emil and Alex we should restore these settings after the
reconfiguration done to support a queue per core.
.John
So the interrupt configuration has to get reset since we have to
assign 2 Tx queues for every Rx queue instead of the 1-1 that was
previously there. That is a natural consequence of rearranging the
queues as currently happens. The issue is the q_vectors themselves
have to be reallocated. The only way to not make that happen would be
to pre-allocate the Tx queues for XDP always.
Also just to be clear we are talking about the interrupts being reset,
not the RSS key right? I just want to make sure that is what we are
talking about.
Yes.
From the tests we did I only observed the IRQs being all reset to the first CPU after Suricata started.
quoted
Thanks.
- Alex
Hi,
We were wondering if there is any follow up/potential solution for that?
If there is something we could help out testing with regards to that
- please let us know.
Thank you
--
Regards,
Peter Manev
We don't have a solution available for this yet. Basically what it
comes down to is that we have to change the driver code so that if
assumes it is going to need to alloc Tx rings for XDP always, and then
if it can't we have to disable the XDP feature. The current logic is
to advertise the XDP feature, and then allocate the rings when XDP is
actually used, and if that fails we fail to load the XDP program.
Unfortunately I don't have an ETA for when we can get to that. It may
be a while, however patches are always welcome.
Thanks.
- Alex
From: Peter Manev <hidden> Date: 2018-02-21 20:01:34
quoted
Hi,
We were wondering if there is any follow up/potential solution for that?
If there is something we could help out testing with regards to that
- please let us know.
Thank you
--
Regards,
Peter Manev
We don't have a solution available for this yet. Basically what it
comes down to is that we have to change the driver code so that if
assumes it is going to need to alloc Tx rings for XDP always, and then
if it can't we have to disable the XDP feature. The current logic is
to advertise the XDP feature, and then allocate the rings when XDP is
actually used, and if that fails we fail to load the XDP program.
Unfortunately I don't have an ETA for when we can get to that. It may
be a while, however patches are always welcome.
Thanks.
- Alex
It seems the issue is not present on 4.15.2 (while it is in 4.13.10
for example) using the ixgbe available in tree
(/lib/modules/4.15.2-amd64/kernel/drivers/net/ethernet/intel/ixgbe)
Not sure what triggered the fix - thought here would be a good place
to ask for some pointers.
Thank you
--
Regards,
Peter Manev