[PATCH] igb: add module param to set max-rss-queues.

From: Ben Greear <redacted>

In systems where you may have a very large number of network
adapters, certain drivers may consume an unfair amount of
IRQ resources.  So, allow a module param that will limit the
number of IRQs at driver load time.  This way, other drivers
(40G Ethernet, for instance), which probably will need the
multiple IRQs more, will not be starved of IRQ resources.

Signed-off-by: Ben Greear <redacted>
---
 drivers/net/ethernet/intel/igb/igb_main.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c
index f9f2874..fdb12e0 100644
--- a/drivers/net/ethernet/intel/igb/igb_main.c
+++ b/drivers/net/ethernet/intel/igb/igb_main.c

@@ -218,6 +218,10 @@ module_param(max_vfs, uint, 0);
 MODULE_PARM_DESC(max_vfs, "Maximum number of virtual functions to allocate per physical function");
 #endif /* CONFIG_PCI_IOV */
 
+static unsigned int max_rss_qs;
+module_param(max_rss_qs, uint, 0);
+MODULE_PARM_DESC(max_rss_qs, "Maximum number of RSS queues.  Forcing lower will use less IRQ resources.");
+
 static pci_ers_result_t igb_io_error_detected(struct pci_dev *,
 		     pci_channel_state_t);
 static pci_ers_result_t igb_io_slot_reset(struct pci_dev *);

@@ -2993,6 +2997,9 @@ static void igb_init_queue_configuration(struct igb_adapter *adapter)
 		break;
 	}
 
+	if (max_rss_qs && max_rss_qs < max_rss_queues)
+		max_rss_queues = max_rss_qs;
+
 	adapter->rss_queues = min_t(u32, max_rss_queues, num_online_cpus());
 
 	igb_set_flag_queue_pairs(adapter, max_rss_queues);

-- 
2.4.11

From: greearb@candelatech.com
Date: Fri, 24 Mar 2017 13:58:47 -0700

From: Ben Greear <redacted>

In systems where you may have a very large number of network
adapters, certain drivers may consume an unfair amount of
IRQ resources.  So, allow a module param that will limit the
number of IRQs at driver load time.  This way, other drivers
(40G Ethernet, for instance), which probably will need the
multiple IRQs more, will not be starved of IRQ resources.

Signed-off-by: Ben Greear <redacted>

Sorry, no module params.

Use generic run-time facilities such as ethtool to configure
such things.

On 03/24/2017 02:12 PM, David Miller wrote:

From: greearb@candelatech.com
Date: Fri, 24 Mar 2017 13:58:47 -0700

quoted

From: Ben Greear <redacted>

In systems where you may have a very large number of network
adapters, certain drivers may consume an unfair amount of
IRQ resources.  So, allow a module param that will limit the
number of IRQs at driver load time.  This way, other drivers
(40G Ethernet, for instance), which probably will need the
multiple IRQs more, will not be starved of IRQ resources.

Signed-off-by: Ben Greear <redacted>

Sorry, no module params.

Use generic run-time facilities such as ethtool to configure
such things.

You cannot call ethtool before module load time, and that is when
the IRQs are first acquired.  It may be way more useful to give each
of 20 network adapters 2 irqs than have the first few grab 16 and the rest
get lumped into legacy crap.

Thanks,
Ben

-- 
Ben Greear [off-list ref]
Candela Technologies Inc  http://www.candelatech.com

From: Ben Greear <redacted>
Date: Fri, 24 Mar 2017 14:20:56 -0700

On 03/24/2017 02:12 PM, David Miller wrote:

quoted

From: greearb@candelatech.com
Date: Fri, 24 Mar 2017 13:58:47 -0700

quoted

From: Ben Greear <redacted>

In systems where you may have a very large number of network
adapters, certain drivers may consume an unfair amount of
IRQ resources.  So, allow a module param that will limit the
number of IRQs at driver load time.  This way, other drivers
(40G Ethernet, for instance), which probably will need the
multiple IRQs more, will not be starved of IRQ resources.

Signed-off-by: Ben Greear <redacted>

Sorry, no module params.

Use generic run-time facilities such as ethtool to configure
such things.

You cannot call ethtool before module load time, and that is when
the IRQs are first acquired.  It may be way more useful to give each
of 20 network adapters 2 irqs than have the first few grab 16 and the
rest
get lumped into legacy crap.

Sorry, you'll have to find a way to fix this without a module
parameter.  I understand it might not be easy, but that's not
a reason to add countless driver private module parameters all
over the tree which is the worst user experience possible.

On Fri, 24 Mar 2017 14:20:56 -0700
Ben Greear [off-list ref] wrote:

On 03/24/2017 02:12 PM, David Miller wrote:

quoted

From: greearb@candelatech.com
Date: Fri, 24 Mar 2017 13:58:47 -0700
 

quoted

From: Ben Greear <redacted>

In systems where you may have a very large number of network
adapters, certain drivers may consume an unfair amount of
IRQ resources.  So, allow a module param that will limit the
number of IRQs at driver load time.  This way, other drivers
(40G Ethernet, for instance), which probably will need the
multiple IRQs more, will not be starved of IRQ resources.

Signed-off-by: Ben Greear <redacted>  

Sorry, no module params.

Use generic run-time facilities such as ethtool to configure
such things.  

You cannot call ethtool before module load time, and that is when
the IRQs are first acquired.  It may be way more useful to give each
of 20 network adapters 2 irqs than have the first few grab 16 and the rest
get lumped into legacy crap.

Almost all network devices do not acquire interrupts until device is brought up.
I.e request_irq is called from open not probe. This is done so that configuration
can be done and also so that unused ports don't consume interrupt space.

On 03/24/2017 04:14 PM, Stephen Hemminger wrote:

On Fri, 24 Mar 2017 14:20:56 -0700
Ben Greear [off-list ref] wrote:

quoted

On 03/24/2017 02:12 PM, David Miller wrote:

quoted

From: greearb@candelatech.com
Date: Fri, 24 Mar 2017 13:58:47 -0700

quoted

From: Ben Greear <redacted>

In systems where you may have a very large number of network
adapters, certain drivers may consume an unfair amount of
IRQ resources.  So, allow a module param that will limit the
number of IRQs at driver load time.  This way, other drivers
(40G Ethernet, for instance), which probably will need the
multiple IRQs more, will not be starved of IRQ resources.

Signed-off-by: Ben Greear <redacted>

Sorry, no module params.

Use generic run-time facilities such as ethtool to configure
such things.

You cannot call ethtool before module load time, and that is when
the IRQs are first acquired.  It may be way more useful to give each
of 20 network adapters 2 irqs than have the first few grab 16 and the rest
get lumped into legacy crap.

Almost all network devices do not acquire interrupts until device is brought up.
I.e request_irq is called from open not probe. This is done so that configuration
can be done and also so that unused ports don't consume interrupt space.

If I ever have to deal with this on stock kernels again I'll keep that in mind.

Thanks,
Ben

-- 
Ben Greear [off-list ref]
Candela Technologies Inc  http://www.candelatech.com

From: greearb@candelatech.com

Sent: 24 March 2017 20:59
From: Ben Greear <redacted>

In systems where you may have a very large number of network
adapters, certain drivers may consume an unfair amount of
IRQ resources.  So, allow a module param that will limit the
number of IRQs at driver load time.  This way, other drivers
(40G Ethernet, for instance), which probably will need the
multiple IRQs more, will not be starved of IRQ resources.

ISTM that the functions for allocating MSI-X should allow a
driver to request additional interrupts after the original probe.

Then drivers than can use additional interrupts but get probed
early won't use up all the vectors before other drivers get
a chance.

	David