Re: [net-next 10/14] net/mlx5: Let user configure io_eq_size param

On Tue, 2021-10-26 at 10:16 -0700, Jakub Kicinski wrote:

On Tue, 26 Oct 2021 15:54:28 +0000 Saeed Mahameed wrote:

quoted

On Tue, 2021-10-26 at 08:05 -0700, Jakub Kicinski wrote:

quoted

On Mon, 25 Oct 2021 13:54:27 -0700 Saeed Mahameed wrote:  

quoted

From: Shay Drory <redacted>

Currently, each I/O EQ is taking 128KB of memory. This size
is not needed in all use cases, and is critical with large
scale.
Hence, allow user to configure the size of I/O EQs.

For example, to reduce I/O EQ size to 64, execute:
$ devlink resource set pci/0000:00:0b.0 path /io_eq_size/ size
64
$ devlink dev reload pci/0000:00:0b.0  

This sort of config is needed by more drivers,
we need a standard way of configuring this.

We had a debate internally about the same thing, Jiri and I thought
that EQ might be a ConnectX only thing (maybe some other vendors
have
it) but it is not really popular

I thought it's a RDMA thing. At least according to grep there's 
a handful of non-MLX drivers which have eqs. Are these not actual
event queues? (huawei/hinic, ibm/ehea, microsoft/mana, qlogic/qed)

quoted

we thought, until other vendors start contributing or asking for 
the same thing, maybe then we can standardize.

Yeah, like the standardization part ever happens :/ 

Look at the EQE/CQE interrupt generation thing. New vendor comes in
and
copies best known practice (which is some driver-specific garbage,
ethtool priv-flags in that case). The maintainer (me) has to be the
policeman remember all those knobs with prior art and push back. Most

Well, i can't even count the patches shot down internally because of
non standard APIs. The driver maintainer (me) has to also be a
policeman. As long as we are in sync I think this can scale, I am sure
other vendors maintainers are filtering as many patches as I do.

of the time the vendor decides to just keep the knob out of tree at
this point, kudos to Hauwei for not giving up. New vendor implements
the API, none of the existing vendors provide reviews or feedback.
Then none of the existing vendors implements the now-standard API.
Someone working for a large customer (me, again) has to go and ask 
for the API to be implemented. Which takes months even tho the
patches
should be trivial.

If the initial patches adding the cqe/eqe interrupt modes to priv-
flags
were nacked and the standard API created we'd all have saved much
time.

Sometimes it is hard to decide what is the best for the user that fits
all vendors, when you are the first to come up with a new concept,
EQE/CQE thing is a relatively new mechanism, took other vendors a while
to catch up, who would've known such mechanism would become popular ?

but I do agree with you  many apis can be standardize or at least
refined with better policing, sorry, but the only way to do this is to
have as many vendors as possible looking at each possible API patch.

Many times we are pioneering in latest features, especially for
smartnics scalability, that involves resource fine tuning and fine-
grained user controls. luckily for the EQ thing we can generalize.

quoted

quoted

Sorry, I didn't have the time to look thru your patches
yesterday, I'm sending a revert for all your new devlink
params.  

Sure, we will submit a RFC to give other vendors a chance to
comment,
it will be basically the same patch (devlink resource) while making
the
parameters vendor generic.

IDK if resource is a right fit (as mentioned to Parav in the
discussion
on the revert).

will switch the discussion to that thread.