quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

Hello Wenzhuo,

I'm testing this patchset, but I am sporadically running
into an issue where the VFs reset fails after the PF flaps.

I have a VM running on a KVM box with a X540-AT2, passing 2 VFs

in.

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

I am using calling rte_eth_dev_reset in response to a
RTE_ETH_EVENT_INTR_RESET callback, and the following
errors appear in the
log:

PMD: ixgbevf_dev_reset(): Ixgbe VF reset: Failed to update link.
PMD: ixgbe_alloc_rx_queue_mbufs(): RX mbuf alloc failed
queue_id=0
PMD: ixgbevf_dev_start(): Unable to initialize RX
hardware
(-12)
PMD: ixgbevf_dev_reset(): Ixgbe VF reset: Failed to start device.

Jumping in with GDB, it seems that the rte_rxmbuf_alloc
call in ixgbe_alloc_rx_queue_mbufs returns NULL at
iteration 64 out of

2048.

quoted

quoted

quoted

quoted

quoted

quoted

The application has ~500 2MB hugepages, and there's 2GB
of free memory available on top of that.

Have you seen this before? Any pointer or suggestion for

debugging?

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

Thanks!

--
Kind regards,
Luca Boccassi

I think the problem is the mbuf occupied by the packets is
not released. This

memory has to be released by the APP, so my patches haven’t
covered

this.

quoted

quoted

quoted

quoted

Actually an example is needed to show how to use the reset API.
I plan to modify the testpmd.

quoted

You may notice this feature is postponed to 16.11. Would
you like to wait for

the new version that will include an example?

Hi,

Unfortunately we need the VF reset working sooner than that,
so one way or the other I'll need to sort it out. Given I've
got a use case where this is happening, if it can be helpful
for you I'm more than happy to help as a guinea pig. If you
could please give some guidance/guidelines with regards to
which API to use to sort the mbuf

problem, I can try it out and give back some feedback.

quoted

quoted

Thanks!

I made a stupid mistake and deleted all my code. So, I have to
take some time to rewrite it :( Attached the example I used to
test the reset API. It's

modified from the l2fwd example. So you can compare it with
l2fwd to see what need to be added.

quoted

Hopefully it can help :)

Thanks! That made me understand a couple of things more, and
I've got past the problem.

Unfortunately now there's a bigger issue - rte_eth_dev_reset is
a blocking

call.

quoted

quoted

the _RESET event callback is fired when the PF goes down, but
when I call rte_eth_dev_reset it will block until the PF goes back up.
There is no way, as far as I can see, to know if the PF is back
up before

calling rte_eth_dev_reset.

quoted

quoted

This is a problem because, as far as I understand, I have to
call all the rte_eth_dev_ APIs from the same thread, in my case
the master thread, and I can't have that block potentially indefinitely.

Would it be possible to have 2 events instead of 1, one when the
PF goes down and one when it goes up? This way an application
would be able to soft-stop the port (drain queues, etc) when the
PF is down, and then call the reset API when it goes back up.

Thanks!

Sorry we cannot have 2 events now. There're 2 problems to have 2 events.
1, Normally we use kernel driver for PF. Now the kernel driver
only have one

kind of message for link down and up. So we cannot tell if it's down or up.

quoted

2, When the PF is down, if we don't reset the VF, VF is not working.
It cannot receive any message from PF. So we cannot know that when
PF is up. It means normally we have to reset VF twice when PF down
and up. (Surely we can wait a while when we receive the message
from PF until PF is up. But we cannot tell how long the time is appropriate.
So this *wait a while* may work for flash.)

Thanks for the clarification, I understand.

The problem with a blocking call is that we basically need to spawn
one thread per rte_eth_dev_reset call, since there is no way of
knowing if a PF is down for good or just flapping, and we can't have
a single thread managing all the interfaces being blocked forever
(EG: PF 1 and 2 go down, thread blocks on PF 1 reset call but it
never returns, meanwhile PF 2 goes back up but call is never made).

A colleague of mine, Eric Kinzie, suggested to add a blocking
boolean parameter to rte_eth_dev_reset API. If set to false, then
the call will not block and just does one try and return an error (EAGAIN ?).

Would this be an acceptable proposition?

quoted

It's a good suggestion.
And I think if the parameter is set to false and the link is not up after trying

once, it will be APP's responsibility to setup a timer or something like that to
keep trying to bring up the link.

That seems reasonable. I've thrown together a quick diff and played with it on
top of your patches and DPDK 2.2, seems to work as intended, I'm attaching it
for reference. Feel free to pick it up, adapt it or ignore it :-)

Also I've noticed that the ixgbe is the only one that actually blocks,
e1000 returns already immediately if the dev_start fails (perhaps it should be
changed to be consistent?) and ixgb40 does weird things that I'm not sure about,
but couldn't spot a loop in there :-)

Also I've used int instead of bool because
drivers/net/e1000/base/e1000_osdep.h redefines bool and true/false, so
compilation fails when including stdbool.h and using bool in rte_ethdev.h

--
Kind regards,
Luca Boccassi

Glad to know it's working now.  Thanks for your patch.  Surely I'll try to include it in the next version :)