From: John Crispin <john@phrozen.org> Date: 2016-06-05 06:33:33
This series contains various small fixes that we stumbled across while
doing thorough testing and code level reviewing of the driver. The only
patch that sticks out is the first one, which addresses a DQL related
issue. The rest are just minor fixes.
John Crispin (12):
net: mediatek: fix DQL support
net: mediatek: add missing return code check
net: mediatek: fix missing free of scratch memory
net: mediatek: invalid buffer lookup in mtk_tx_map()
net: mediatek: dropped rx packets are not being counted properly
net: mediatek: add next data pointer coherency protection
net: mediatek: disable all interrupts during probe
net: mediatek: fix threshold value
net: mediatek: increase watchdog_timeo
net: mediatek: fix off by one in the TX ring allocation
net: mediatek: only wake the queue if it is stopped
net: mediatek: remove superfluous queue wake up call
drivers/net/ethernet/mediatek/mtk_eth_soc.c | 94 ++++++++++++++++++---------
drivers/net/ethernet/mediatek/mtk_eth_soc.h | 3 +
2 files changed, 65 insertions(+), 32 deletions(-)
--
1.7.10.4
From: John Crispin <john@phrozen.org> Date: 2016-06-05 06:33:41
Scratch memory gets allocated in mtk_init_fq_dma() but the corresponding
code to free it is missing inside mtk_dma_free() causing a memory leak.
With this patch applied, we can run ifconfig up/down several thousand
times without any problems.
Signed-off-by: John Crispin <john@phrozen.org>
---
drivers/net/ethernet/mediatek/mtk_eth_soc.c | 18 +++++++++++++-----
drivers/net/ethernet/mediatek/mtk_eth_soc.h | 2 ++
2 files changed, 15 insertions(+), 5 deletions(-)
From: John Crispin <john@phrozen.org> Date: 2016-06-05 06:33:56
There are two places inside mtk_poll_rx where rx_dropped is not being
incremented properly. Fix this by adding the missing code to increment
the counter.
Signed-off-by: John Crispin <redacted>
---
drivers/net/ethernet/mediatek/mtk_eth_soc.c | 2 ++
1 file changed, 2 insertions(+)
From: John Crispin <john@phrozen.org> Date: 2016-06-05 06:33:58
During stress testing, after reducing the threshold value, we have seen
TX timeouts that were caused by the watchdog_timeo value being too low.
Increase the value to 5 * HZ which is a value commonly used by many other
drivers.
Signed-off-by: John Crispin <redacted>
---
drivers/net/ethernet/mediatek/mtk_eth_soc.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
From: John Crispin <john@phrozen.org> Date: 2016-06-05 06:34:00
The current code only disables those IRQs that we will later use. To
ensure that we have a predefined state, we really want to disable all IRQs.
Change the code to disable all IRQs to achieve this.
Signed-off-by: John Crispin <redacted>
---
drivers/net/ethernet/mediatek/mtk_eth_soc.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
From: John Crispin <john@phrozen.org> Date: 2016-06-05 06:34:02
The logic to calculate the threshold value for stopping the TX queue is
bad. Currently it will always use 1/2 of the rings size, which is way too
much. Set the threshold to MAX_SKB_FRAGS. This makes sure that the queue
is stopped when there is not enough room to accept an additional segment.
Signed-off-by: John Crispin <john@phrozen.org>
---
drivers/net/ethernet/mediatek/mtk_eth_soc.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
@@ -1012,8 +1012,7 @@ static int mtk_tx_alloc(struct mtk_eth *eth)atomic_set(&ring->free_count,MTK_DMA_SIZE-2);ring->next_free=&ring->dma[0];ring->last_free=&ring->dma[MTK_DMA_SIZE-2];-ring->thresh=max((unsignedlong)MTK_DMA_SIZE>>2,-MAX_SKB_FRAGS);+ring->thresh=MAX_SKB_FRAGS;/* make sure that all changes to the dma ring are flushed before we*continue
--
1.7.10.4
_______________________________________________
Linux-mediatek mailing list
Linux-mediatek@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-mediatek
From: John Crispin <john@phrozen.org> Date: 2016-06-05 06:34:05
The MTK ethernet core has 2 MACs both sitting on the same DMA ring. For
DQL to be deterministic it needs to track the amount of data in the DMA
ring and not the amount of data enqueued on each device. The current code
is incorrect, fix it by making it each device track its own traffic aswell
as the traffic of the other device.
Signed-off-by: John Crispin <redacted>
---
drivers/net/ethernet/mediatek/mtk_eth_soc.c | 33 ++++++++++++++++-----------
1 file changed, 20 insertions(+), 13 deletions(-)
@@ -625,7 +625,16 @@ static int mtk_tx_map(struct sk_buff *skb, struct net_device *dev,WRITE_ONCE(itxd->txd3,(TX_DMA_SWC|TX_DMA_PLEN0(skb_headlen(skb))|(!nr_frags*TX_DMA_LS0)));-netdev_sent_queue(dev,skb->len);+/* we have a single DMA ring so BQL needs to be updated for all devices+*sittingonthisring+*/+for(i=0;i<MTK_MAC_COUNT;i++){+if(!eth->netdev[i])+continue;++netdev_sent_queue(eth->netdev[i],skb->len);+}+skb_tx_timestamp(skb);ring->next_free=mtk_qdma_phys_to_virt(ring,txd->txd2);
@@ -853,21 +862,18 @@ static int mtk_poll_tx(struct mtk_eth *eth, int budget, bool *tx_again)structmtk_tx_dma*desc;structsk_buff*skb;structmtk_tx_buf*tx_buf;-inttotal=0,done[MTK_MAX_DEVS];-unsignedintbytes[MTK_MAX_DEVS];+inttotal=0,done=0;+unsignedintbytes=0;u32cpu,dma;staticintcondition;inti;-memset(done,0,sizeof(done));-memset(bytes,0,sizeof(bytes));-cpu=mtk_r32(eth,MTK_QTX_CRX_PTR);dma=mtk_r32(eth,MTK_QTX_DRX_PTR);desc=mtk_qdma_phys_to_virt(ring,cpu);-while((cpu!=dma)&&budget){+while((cpu!=dma)&&done<budget){u32next_cpu=desc->txd2;intmac;
@@ -887,9 +893,8 @@ static int mtk_poll_tx(struct mtk_eth *eth, int budget, bool *tx_again)}if(skb!=(structsk_buff*)MTK_DMA_DUMMY_DESC){-bytes[mac]+=skb->len;-done[mac]++;-budget--;+bytes+=skb->len;+done++;}mtk_tx_unmap(eth->dev,tx_buf);
@@ -902,11 +907,13 @@ static int mtk_poll_tx(struct mtk_eth *eth, int budget, bool *tx_again)mtk_w32(eth,cpu,MTK_QTX_CRX_PTR);+/* we have a single DMA ring so BQL needs to be updated for all devices+*sittingonthisring+*/for(i=0;i<MTK_MAC_COUNT;i++){-if(!eth->netdev[i]||!done[i])+if(!eth->netdev[i])continue;-netdev_completed_queue(eth->netdev[i],done[i],bytes[i]);-total+=done[i];+netdev_completed_queue(eth->netdev[i],done,bytes);}/* read hw index again make sure no new tx packet */
From: John Crispin <john@phrozen.org> Date: 2016-06-05 06:34:09
The code fails to check if the scratch memory was properly allocated. Add
this check and return with an error if the allocation failed.
Signed-off-by: John Crispin <redacted>
---
drivers/net/ethernet/mediatek/mtk_eth_soc.c | 3 +++
1 file changed, 3 insertions(+)
From: John Crispin <john@phrozen.org> Date: 2016-06-05 06:35:49
The QDMA engine can fail to update the register pointing to the next TX
descriptor if this bit does not get set in the QDMA configuration register.
Not setting this bit can result in invalid values inside the TX rings
registers which will causes TX stalls.
Signed-off-by: John Crispin <redacted>
---
drivers/net/ethernet/mediatek/mtk_eth_soc.c | 2 +-
drivers/net/ethernet/mediatek/mtk_eth_soc.h | 1 +
2 files changed, 2 insertions(+), 1 deletion(-)
From: John Crispin <john@phrozen.org> Date: 2016-06-05 06:36:12
The lookup of the tx_buffer in the error path inside mtk_tx_map() uses the
wrong descriptor pointer. This looks like a copy & paste error. Change the
code to use the correct pointer.
Signed-off-by: John Crispin <redacted>
---
drivers/net/ethernet/mediatek/mtk_eth_soc.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
From: John Crispin <john@phrozen.org> Date: 2016-06-05 06:59:55
The code checks if the queue should be stopped because we are below the
threshold of free descriptors only to check if it should be started again.
If we do end up in a state where we are at the threshold limit, it makes
more sense to just stop the queue and wait for the next IRQ to trigger the
TX housekeeping again. There is no rush in enqueuing the next packet, it
needs to wait for all the others in the queue to be dispatched first
anyway.
Signed-off-by: John Crispin <john@phrozen.org>
---
drivers/net/ethernet/mediatek/mtk_eth_soc.c | 7 ++-----
1 file changed, 2 insertions(+), 5 deletions(-)
From: John Crispin <john@phrozen.org> Date: 2016-06-05 07:00:25
The current code unconditionally wakes up the queue at the end of each
tx_poll action. Change the code to only wake up the queues if any of
them have actually been stopped before.
Signed-off-by: John Crispin <john@phrozen.org>
---
drivers/net/ethernet/mediatek/mtk_eth_soc.c | 17 ++++++++++++++++-
1 file changed, 16 insertions(+), 1 deletion(-)
From: John Crispin <john@phrozen.org> Date: 2016-06-05 07:00:53
The TX ring setup has an off by one error causing it to not utilise all
descriptors. This has the side effect that we need to reset the next
pointer at runtime to make it work. Fix the off by one and remove the
code fixing the ring at runtime.
Signed-off-by: John Crispin <john@phrozen.org>
---
drivers/net/ethernet/mediatek/mtk_eth_soc.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
@@ -903,7 +903,6 @@ static int mtk_poll_tx(struct mtk_eth *eth, int budget, bool *tx_again)}mtk_tx_unmap(eth->dev,tx_buf);-ring->last_free->txd2=next_cpu;ring->last_free=desc;atomic_inc(&ring->free_count);
@@ -1011,7 +1010,7 @@ static int mtk_tx_alloc(struct mtk_eth *eth)atomic_set(&ring->free_count,MTK_DMA_SIZE-2);ring->next_free=&ring->dma[0];-ring->last_free=&ring->dma[MTK_DMA_SIZE-2];+ring->last_free=&ring->dma[MTK_DMA_SIZE-1];ring->thresh=MAX_SKB_FRAGS;/* make sure that all changes to the dma ring are flushed before we
From: David Miller <davem@davemloft.net> Date: 2016-06-05 07:32:50
From: John Crispin <john@phrozen.org>
Date: Sun, 5 Jun 2016 08:32:54 +0200
quoted hunk
@@ -625,7 +625,16 @@ static int mtk_tx_map(struct sk_buff *skb, struct net_device *dev, WRITE_ONCE(itxd->txd3, (TX_DMA_SWC | TX_DMA_PLEN0(skb_headlen(skb)) | (!nr_frags * TX_DMA_LS0)));- netdev_sent_queue(dev, skb->len);+ /* we have a single DMA ring so BQL needs to be updated for all devices+ * sitting on this ring+ */+ for (i = 0; i < MTK_MAC_COUNT; i++) {+ if (!eth->netdev[i])+ continue;++ netdev_sent_queue(eth->netdev[i], skb->len);+ }+ skb_tx_timestamp(skb);
Sorry, this is very far from working.
You cannot asynchronously touch the DQL state of another netdevice.
You have to hold the TX lock of a queue while changing it's DQL state,
otherwise you'll corrupt the state.
This "loop over all possible devices on this DMA ring" is pretty
expensive for the problem you're trying to solve.
You'll have to find another way to fix this bug, which BTW I'm not too
clear about. The commit message doesn't explain sufficiently what the
actual problem is. "not deterministic" doesn't give enough details.
From: Andrew Lunn <andrew@lunn.ch> Date: 2016-06-05 14:56:46
On Sun, Jun 05, 2016 at 08:33:02AM +0200, John Crispin wrote:
During stress testing, after reducing the threshold value, we have seen
TX timeouts that were caused by the watchdog_timeo value being too low.
Increase the value to 5 * HZ which is a value commonly used by many other
drivers.
I've never studied what watchdog_timeo actually means. Does it mean a
transmit has not completed in that amount of time? Would this imply
you have 5 seconds worth of packets in your transmit queue? Do you
know what the driver is doing during this 5 seconds?
Thanks
Andrew
From: John Crispin <john@phrozen.org> Date: 2016-06-06 06:24:24
On 05/06/2016 16:56, Andrew Lunn wrote:
On Sun, Jun 05, 2016 at 08:33:02AM +0200, John Crispin wrote:
quoted
During stress testing, after reducing the threshold value, we have seen
TX timeouts that were caused by the watchdog_timeo value being too low.
Increase the value to 5 * HZ which is a value commonly used by many other
drivers.
I've never studied what watchdog_timeo actually means. Does it mean a
transmit has not completed in that amount of time? Would this imply
you have 5 seconds worth of packets in your transmit queue? Do you
know what the driver is doing during this 5 seconds?
Hi Andrew,
it is waiting for the watchdog to trigger :-) TBH the 1s seems to be too
short to for the dma ring length to be flushed and i had to pick some
value and 5 is used most places.
it really depends on the amount of packets in the queue, their length
and the mac setting. the timeout needs to be large enough that it would
not trigger incorrectly even if the mac is on 10mbit half duplex and all
frames in the queue were maximum size.
John
From: John Crispin <john@phrozen.org> Date: 2016-06-06 06:43:21
On 05/06/2016 09:32, David Miller wrote:
From: John Crispin <john@phrozen.org>
Date: Sun, 5 Jun 2016 08:32:54 +0200
quoted
@@ -625,7 +625,16 @@ static int mtk_tx_map(struct sk_buff *skb, struct net_device *dev, WRITE_ONCE(itxd->txd3, (TX_DMA_SWC | TX_DMA_PLEN0(skb_headlen(skb)) | (!nr_frags * TX_DMA_LS0)));- netdev_sent_queue(dev, skb->len);+ /* we have a single DMA ring so BQL needs to be updated for all devices+ * sitting on this ring+ */+ for (i = 0; i < MTK_MAC_COUNT; i++) {+ if (!eth->netdev[i])+ continue;++ netdev_sent_queue(eth->netdev[i], skb->len);+ }+ skb_tx_timestamp(skb);
Sorry, this is very far from working.
You cannot asynchronously touch the DQL state of another netdevice.
You have to hold the TX lock of a queue while changing it's DQL state,
otherwise you'll corrupt the state.
This "loop over all possible devices on this DMA ring" is pretty
expensive for the problem you're trying to solve.
You'll have to find another way to fix this bug, which BTW I'm not too
clear about. The commit message doesn't explain sufficiently what the
actual problem is. "not deterministic" doesn't give enough details.
Hi David,
DQL is supposed to measure how much data is enqueued on a netdev. the
problem here is that two devices share the same hardware queue. fq_codel
for example uses the values from dql to base its QoS judgement on. if we
track the dql of the 2 devices separately then the values will not take
the actual amount of data enqueued into account but only parts of it.
this will make the queue length used as a basis for fq_codel
calculations non deterministic thus breaking qos. dql needs to track the
amount of data in the physical queue underlying the netdev to be useful.
hope that explanation is better to understand.
i think one solution would be to add some code to have 2 devices share
the same dql instance. would that be an acceptable solution ?
anyhow, i will resend the series without the dql patch today and then
worry about it afterwards.
John
From: Andrew Lunn <andrew@lunn.ch> Date: 2016-06-06 12:21:38
Hi Andrew,
it is waiting for the watchdog to trigger :-) TBH the 1s seems to be too
short to for the dma ring length to be flushed and i had to pick some
value and 5 is used most places.
it really depends on the amount of packets in the queue, their length
and the mac setting. the timeout needs to be large enough that it would
not trigger incorrectly even if the mac is on 10mbit half duplex and all
frames in the queue were maximum size.
So you are saying there is 5 seconds worth of traffic in the transmit ring.
As a general point, not specific to this driver, is that wise? Isn't
that really bad buffer bloat?
I just wondered what happened to cause it to have 5 seconds worth of
traffic in the transmit ring. Did downstream signal a pause? But i
thought the byte queue limit was designed to prevent a big backlog in
the transmit queue? At 10/Half, is it not reacting fast enough? Since
it is half duplex, do you have a lot of traffic coming the other way
and something is not being fair at distributing up and down traffic?
I'm just wondering if by increasing the watchdog to 5 seconds, you are
just hiding a problem.
Andrew
From: John Crispin <john@phrozen.org> Date: 2016-06-06 12:38:56
On 06/06/2016 14:21, Andrew Lunn wrote:
quoted
Hi Andrew,
it is waiting for the watchdog to trigger :-) TBH the 1s seems to be too
short to for the dma ring length to be flushed and i had to pick some
value and 5 is used most places.
it really depends on the amount of packets in the queue, their length
and the mac setting. the timeout needs to be large enough that it would
not trigger incorrectly even if the mac is on 10mbit half duplex and all
frames in the queue were maximum size.
So you are saying there is 5 seconds worth of traffic in the transmit ring.
As a general point, not specific to this driver, is that wise? Isn't
that really bad buffer bloat?
I just wondered what happened to cause it to have 5 seconds worth of
traffic in the transmit ring. Did downstream signal a pause? But i
thought the byte queue limit was designed to prevent a big backlog in
the transmit queue? At 10/Half, is it not reacting fast enough? Since
it is half duplex, do you have a lot of traffic coming the other way
and something is not being fair at distributing up and down traffic?
I'm just wondering if by increasing the watchdog to 5 seconds, you are
just hiding a problem.
Andrew
Hi Andrew,
running the driver without any QoS and using the typical ringsize for
gigabit devices, 1s is not enough. we were seeing false positive
watchdog events. then i grepped to see what other drivers do and most
set 5seconds. ideally the watchdog never triggers as the driver is
functional an does not suffer from deadlocks.
at gbit ethernet can transmit 83 packets that are 1500 bytes long /
second, if there are no pause gaps. at 10Mbit that would be 6 packets.
so assuming we have a ring of 128 and napi set to 64, we would want at
least 2 seconds, 3 if there are a lot of pause gaps, 4-5 if it is half
duplex ... so i took the value commonly used, which is 5 according to grep.
figuring out when the queue is stuck seems to be a little bit more
complicated. imho the trigger should not be based on how long it took to
send a packet, but how long since the last packet was dequeued from dma.
personally i'd rather fix the deadlocks that can happen, which is what
we did, than rely on the watchdog to reset the queue. right now we can
hammer the driver with several streams on both macs utilizing all 4 cpu
cores for several days without seeing any hickups.
John
From: David Miller <davem@davemloft.net> Date: 2016-06-07 23:01:19
From: John Crispin <john@phrozen.org>
Date: Mon, 6 Jun 2016 08:43:13 +0200
i think one solution would be to add some code to have 2 devices share
the same dql instance. would that be an acceptable solution ?
You still need to address the issue of synchronization.
dql purposefully doesn't use locking, always because a higher level
object (in this case the netdev TX queue) it is contained within
provides the synchronization.
That breaks apart once you share the dql between two netdevs, as you
are proposing here. You'll have to add locking, which is expensive.
That's why I'm trying to encourage you to think out of the box and
find some way to solve the issue without having to access shared
state shared between multiple devices.
Thanks.
From: Tom Herbert <hidden> Date: 2016-06-07 23:20:36
On Tue, Jun 7, 2016 at 4:01 PM, David Miller [off-list ref] wrote:
From: John Crispin <john@phrozen.org>
Date: Mon, 6 Jun 2016 08:43:13 +0200
quoted
i think one solution would be to add some code to have 2 devices share
the same dql instance. would that be an acceptable solution ?
You still need to address the issue of synchronization.
dql purposefully doesn't use locking, always because a higher level
object (in this case the netdev TX queue) it is contained within
provides the synchronization.
That breaks apart once you share the dql between two netdevs, as you
are proposing here. You'll have to add locking, which is expensive.
That's why I'm trying to encourage you to think out of the box and
find some way to solve the issue without having to access shared
state shared between multiple devices.
I think you guys mean mean BQL not DQL :-)
If two netdevs share the same DMA ring then is using two netdevs the
right approach. Seems like this would have other consequences beyond
BQL...
Tom