Re: [RFC PATCH net] net: dsa: flush switchdev workqueue before tearing down CPU/DSA ports

On Sun, Sep 12, 2021 at 07:20:23PM -0700, Florian Fainelli wrote:

On 9/12/2021 7:12 PM, Vladimir Oltean wrote:

quoted

On Sun, Sep 12, 2021 at 07:06:25PM -0700, Florian Fainelli wrote:

quoted

On 9/12/2021 9:33 AM, Vladimir Oltean wrote:

quoted

On Sun, Sep 12, 2021 at 09:24:53AM -0700, Florian Fainelli wrote:

quoted

On 9/12/2021 9:19 AM, Vladimir Oltean wrote:

quoted

On Sun, Sep 12, 2021 at 09:13:36AM -0700, Florian Fainelli wrote:

quoted

Reviewed-by: Florian Fainelli <f.fainelli@gmail.com>

Did you post this as a RFC for a particular reason, or just to give
reviewers some time?

Both.

In principle there's nothing wrong with what this patch does, only
perhaps maybe something with what it doesn't do.

We keep saying that a network interface should be ready to pass traffic
as soon as it's registered, but that "walk dst->ports linearly when
calling dsa_port_setup" might not really live up to that promise.

That promise most definitively existed back when Lennert wrote this code and
we had an array of ports and the switch drivers brought up their port in
their ->setup() method, nowadays, not so sure anymore because of the
.port_enable() as much as the list.

This is making me wonder whether the occasional messages I am seeing on
system suspend from __dev_queue_xmit: Virtual device %s asks to queue
packet! might have something to do with that and/or the inappropriate
ordering between suspending the switch and the DSA master.

Sorry, I have never tested the suspend/resume code path, mostly because
I don't know what would the easiest way be to wake up my systems from
suspend. If you could give me some pointers there I would be glad to
look into it.

If your systems support suspend/resume just do:

echo mem > /sys/power/state
or
echo standby > /sys/power/state

if they don't, then maybe a x86 VM with dsa_loop may precipitate the
problem, but since it uses DSA_TAG_PROTO_NONE, I doubt it, we would need to
pass traffic on the DSA devices for this warning to show up.

I figured out a working combination in the meanwhile, I even found a bug
in the process:
https://patchwork.kernel.org/project/netdevbpf/patch/20210912192805.1394305-1-vladimir.oltean@nxp.com/

However I did not see those messages getting printed while pinging after
system resume (note that none of the DSA switch drivers I tested with
did implement .suspend or .resume), with net-next or with linux-stable/linux-5.14.y.

Is there more to your setup to reproduce this issue?

All switch ports are brought up with a DHCP client, the issue is
definitively intermittent and not frequent, I don't have suspend/resume
working on 5.14.y yet, but going as far back as 5.10.y should let you see
the same kind of messages.

Nope, I don't see the message on linux-5.10.y either, but even if I look
at the code I don't understand what goes on.

Here are some facts:

- __dev_queue_xmit only prints "Virtual device %s asks to queue packet!"
  for devices which have a NULL dev->qdisc->enqueue, but return
  NETDEV_TX_BUSY in dev_hard_start_xmit

- only the noqueue qdisc manages that performance, normally register_qdisc
  sets up a default of ->enqueue pointer of noop_qdisc_ops.enqueue, but
  there is a hack in noqueue_init which subverts that.

- The default qdiscs are assigned by attach_default_qdiscs(). There are
  two ways in which an interface can get the noqueue_qdisc_ops by default:
  (a) it declares the IFF_NO_QUEUE feature
  (b) the init of its other default qdisc (for example mq) failed for
      some reason, the kernel prints a warning and falls back to noqueue.

- DSA declares IFF_NO_QUEUE, so it uses the noqueue qdisc by default

- DSA never returns anything other than NETDEV_TX_OK in ndo_start_xmit.

So my conclusion is:
you have the noqueue qdisc on the DSA master, which got there either by
accident due to a failure of the mq qdisc initialization, or
intentionally by:

tc qdisc add dev eth0 root noqueue

When you use the noqueue qdisc and your DSA master device driver returns
NETDEV_TX_BUSY, the packet won't get queued => the system pretty much is
in its own right to complain.