Hi,
The SW eventdev rx adapter has an internal enqueue buffer 'rx_adapter->event_enqueue_buffer', which stores packets received from the NIC until at least BATCH_SIZE (=32) packets have been received before enqueueing them to eventdev. For example in case of validation testing, where often a small number of specific test packets is sent to the NIC, this causes a lot of problems. One would always have to transmit at least BATCH_SIZE test packets before anything can be received from eventdev. Additionally, if the rx packet rate is slow this also adds a considerable amount of additional delay.
Looking at the rx adapter API and sw implementation code there doesn’t seem to be a way to disable this internal caching. In my opinion this “functionality" makes testing sw rx adapter so cumbersome that either the implementation should be modified to enqueue the cached packets after a while (some performance penalty) or there should be some method to disable caching. Any opinions how this issue could be fixed?
Regards,
Matias
Hi,
The SW eventdev rx adapter has an internal enqueue buffer 'rx_adapter-
quoted
event_enqueue_buffer', which stores packets received from the NIC until at
least BATCH_SIZE (=32) packets have been received before enqueueing them
to eventdev. For example in case of validation testing, where often a small
number of specific test packets is sent to the NIC, this causes a lot of
problems. One would always have to transmit at least BATCH_SIZE test
packets before anything can be received from eventdev. Additionally, if the rx
packet rate is slow this also adds a considerable amount of additional delay.
Looking at the rx adapter API and sw implementation code there doesn’t
seem to be a way to disable this internal caching. In my opinion this
“functionality" makes testing sw rx adapter so cumbersome that either the
implementation should be modified to enqueue the cached packets after a
while (some performance penalty) or there should be some method to
disable caching. Any opinions how this issue could be fixed?
At the minimum, I would think there should be a compile time option.
From a use case perspective, I think it falls under latency vs throughput considerations. If there is a latency sensitive application, it might not want to wait till 32 packets are received.
On 2019-05-07 11:52, Elo, Matias (Nokia - FI/Espoo) wrote:
Hi,
The SW eventdev rx adapter has an internal enqueue buffer 'rx_adapter->event_enqueue_buffer', which stores packets received from the NIC until at least BATCH_SIZE (=32) packets have been received before enqueueing them to eventdev. For example in case of validation testing, where often a small number of specific test packets is sent to the NIC, this causes a lot of problems. One would always have to transmit at least BATCH_SIZE test packets before anything can be received from eventdev. Additionally, if the rx packet rate is slow this also adds a considerable amount of additional delay.
Looking at the rx adapter API and sw implementation code there doesn’t seem to be a way to disable this internal caching. In my opinion this “functionality" makes testing sw rx adapter so cumbersome that either the implementation should be modified to enqueue the cached packets after a while (some performance penalty) or there should be some method to disable caching. Any opinions how this issue could be fixed?
The rx adaptor's service function will be called repeatedly, at a very high frequency (especially in near-idle situations). One potential scheme is to, by means of a counter, keeping track of the number of calls since the last packet was received from the NIC, and flush the buffers a number of idle (zero-NIC-dequeue) calls.
In that case, you maintain good performance, while not introducing too much latency.
The DSW Event Device takes this approach to flushing its internal buffers.
Another way would be to use a timer. Either an adapter-internal TSC timestamp for buffer age, or a rte_timer timer. rdtsc is not for free, so I would lean toward the first option.
Hi,
The SW eventdev rx adapter has an internal enqueue buffer 'rx_adapter-
quoted
event_enqueue_buffer', which stores packets received from the NIC until at
least BATCH_SIZE (=32) packets have been received before enqueueing them
to eventdev. For example in case of validation testing, where often a small
number of specific test packets is sent to the NIC, this causes a lot of
problems. One would always have to transmit at least BATCH_SIZE test
packets before anything can be received from eventdev. Additionally, if the rx
packet rate is slow this also adds a considerable amount of additional delay.
Looking at the rx adapter API and sw implementation code there doesn’t
seem to be a way to disable this internal caching. In my opinion this
“functionality" makes testing sw rx adapter so cumbersome that either the
implementation should be modified to enqueue the cached packets after a
while (some performance penalty) or there should be some method to
disable caching. Any opinions how this issue could be fixed?
At the minimum, I would think there should be a compile time option.
From a use case perspective, I think it falls under latency vs throughput considerations. If there is a latency sensitive application, it might not want to wait till 32 packets are received.
From what I understood from Matias Elo and also after a quick glance in the code, the unlucky packets will be buffered indefinitely, in case the system goes idle. This is totally unacceptable (both in production and validation), in my opinion, and should be filed as a bug.
On 7 May 2019, at 15:01, Mattias Rönnblom <hofors@lysator.liu.se<mailto:hofors@lysator.liu.se>> wrote:
On 2019-05-07 13:12, Honnappa Nagarahalli wrote:
Hi,
The SW eventdev rx adapter has an internal enqueue buffer 'rx_adapter-
event_enqueue_buffer', which stores packets received from the NIC until at
least BATCH_SIZE (=32) packets have been received before enqueueing them
to eventdev. For example in case of validation testing, where often a small
number of specific test packets is sent to the NIC, this causes a lot of
problems. One would always have to transmit at least BATCH_SIZE test
packets before anything can be received from eventdev. Additionally, if the rx
packet rate is slow this also adds a considerable amount of additional delay.
Looking at the rx adapter API and sw implementation code there doesn’t
seem to be a way to disable this internal caching. In my opinion this
“functionality" makes testing sw rx adapter so cumbersome that either the
implementation should be modified to enqueue the cached packets after a
while (some performance penalty) or there should be some method to
disable caching. Any opinions how this issue could be fixed?
At the minimum, I would think there should be a compile time option.
From a use case perspective, I think it falls under latency vs throughput considerations. If there is a latency sensitive application, it might not want to wait till 32 packets are received.
From what I understood from Matias Elo and also after a quick glance in the code, the unlucky packets will be buffered indefinitely, in case the system goes idle. This is totally unacceptable (both in production and validation), in my opinion, and should be filed as a bug.
Indeed, this is what happens. I’ll create a bug report to track this issue.
From: Jerin Jacob Kollanukkaran <hidden> Date: 2019-05-07 12:14:06
+ Nikhil
Please add respective maintainer from MAINTAINERS file for quick resolution.
-----Original Message-----
From: dev <redacted> On Behalf Of Elo, Matias (Nokia -
FI/Espoo)
Sent: Tuesday, May 7, 2019 5:33 PM
To: Mattias Rönnblom <redacted>
Cc: Honnappa Nagarahalli <redacted>; dev@dpdk.org;
nd [off-list ref]
Subject: Re: [dpdk-dev] eventdev: sw rx adapter enqueue caching
On 7 May 2019, at 15:01, Mattias Rönnblom
<hofors@lysator.liu.se<mailto:hofors@lysator.liu.se>> wrote:
On 2019-05-07 13:12, Honnappa Nagarahalli wrote:
Hi,
The SW eventdev rx adapter has an internal enqueue buffer 'rx_adapter-
event_enqueue_buffer', which stores packets received from the NIC until at
least BATCH_SIZE (=32) packets have been received before enqueueing them to
eventdev. For example in case of validation testing, where often a small number
of specific test packets is sent to the NIC, this causes a lot of problems. One
would always have to transmit at least BATCH_SIZE test packets before anything
can be received from eventdev. Additionally, if the rx packet rate is slow this
also adds a considerable amount of additional delay.
Looking at the rx adapter API and sw implementation code there doesn’t seem
to be a way to disable this internal caching. In my opinion this “functionality"
makes testing sw rx adapter so cumbersome that either the implementation
should be modified to enqueue the cached packets after a while (some
performance penalty) or there should be some method to disable caching. Any
opinions how this issue could be fixed?
At the minimum, I would think there should be a compile time option.
From a use case perspective, I think it falls under latency vs throughput
considerations. If there is a latency sensitive application, it might not want to
wait till 32 packets are received.
From what I understood from Matias Elo and also after a quick glance in the
code, the unlucky packets will be buffered indefinitely, in case the system goes
idle. This is totally unacceptable (both in production and validation), in my
opinion, and should be filed as a bug.
Indeed, this is what happens. I’ll create a bug report to track this issue.
-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Elo, Matias (Nokia -
FI/Espoo)
Sent: Tuesday, May 7, 2019 5:33 PM
To: Mattias Rönnblom <redacted>
Cc: Honnappa Nagarahalli <redacted>;
dev@dpdk.org; nd [off-list ref]
Subject: Re: [dpdk-dev] eventdev: sw rx adapter enqueue caching
On 7 May 2019, at 15:01, Mattias Rönnblom
<hofors@lysator.liu.se<mailto:hofors@lysator.liu.se>> wrote:
On 2019-05-07 13:12, Honnappa Nagarahalli wrote:
Hi,
The SW eventdev rx adapter has an internal enqueue buffer 'rx_adapter-
event_enqueue_buffer', which stores packets received from the NIC until at
least BATCH_SIZE (=32) packets have been received before enqueueing them
to eventdev. For example in case of validation testing, where often a small
number of specific test packets is sent to the NIC, this causes a lot of
problems. One would always have to transmit at least BATCH_SIZE test
packets before anything can be received from eventdev. Additionally, if the
rx
packet rate is slow this also adds a considerable amount of additional delay.
Looking at the rx adapter API and sw implementation code there doesn’t
seem to be a way to disable this internal caching. In my opinion this
“functionality" makes testing sw rx adapter so cumbersome that either the
implementation should be modified to enqueue the cached packets after a
while (some performance penalty) or there should be some method to
disable caching. Any opinions how this issue could be fixed?
At the minimum, I would think there should be a compile time option.
From a use case perspective, I think it falls under latency vs throughput
considerations. If there is a latency sensitive application, it might not want
to wait till 32 packets are received.
From what I understood from Matias Elo and also after a quick glance in the
code, the unlucky packets will be buffered indefinitely, in case the system
goes idle. This is totally unacceptable (both in production and validation), in
my opinion, and should be filed as a bug.
Indeed, this is what happens. I’ll create a bug report to track this issue.
Thanks, I’ve tested this patch and can confirm that it fixes the problem.
I didn’t do any performance comparison but at least with high packet rate rte_eth_rx_burst() should already return close to BATCH_SIZE packets, so the performance hit shouldn’t be that big.
-Matias
On 9 May 2019, at 14:24, Rao, Nikhil <nikhil.rao@intel.com<mailto:nikhil.rao@intel.com>> wrote:
-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Elo, Matias (Nokia -
FI/Espoo)
Sent: Tuesday, May 7, 2019 5:33 PM
To: Mattias Rönnblom <hofors@lysator.liu.se<mailto:hofors@lysator.liu.se>>
Cc: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com<mailto:Honnappa.Nagarahalli@arm.com>>;
dev@dpdk.org<mailto:dev@dpdk.org>; nd <nd@arm.com<mailto:nd@arm.com>>
Subject: Re: [dpdk-dev] eventdev: sw rx adapter enqueue caching
On 7 May 2019, at 15:01, Mattias Rönnblom
<hofors@lysator.liu.se<mailto:hofors@lysator.liu.se><mailto:hofors@lysator.liu.se>> wrote:
On 2019-05-07 13:12, Honnappa Nagarahalli wrote:
Hi,
The SW eventdev rx adapter has an internal enqueue buffer 'rx_adapter-
event_enqueue_buffer', which stores packets received from the NIC until at
least BATCH_SIZE (=32) packets have been received before enqueueing them
to eventdev. For example in case of validation testing, where often a small
number of specific test packets is sent to the NIC, this causes a lot of
problems. One would always have to transmit at least BATCH_SIZE test
packets before anything can be received from eventdev. Additionally, if the
rx
packet rate is slow this also adds a considerable amount of additional delay.
Looking at the rx adapter API and sw implementation code there doesn’t
seem to be a way to disable this internal caching. In my opinion this
“functionality" makes testing sw rx adapter so cumbersome that either the
implementation should be modified to enqueue the cached packets after a
while (some performance penalty) or there should be some method to
disable caching. Any opinions how this issue could be fixed?
At the minimum, I would think there should be a compile time option.
From a use case perspective, I think it falls under latency vs throughput
considerations. If there is a latency sensitive application, it might not want
to wait till 32 packets are received.
From what I understood from Matias Elo and also after a quick glance in the
code, the unlucky packets will be buffered indefinitely, in case the system
goes idle. This is totally unacceptable (both in production and validation), in
my opinion, and should be filed as a bug.
Indeed, this is what happens. I’ll create a bug report to track this issue.
I have posted a patch for this issue
http://patchwork.dpdk.org/patch/53350/
Please let me know your comments.
Thanks,
Nikhil