net: r8169: a question of memory barrier in the r8169 driver

In the rt8169 driver, the function "rtl_tx" uses "smp_mb" to sync the 
writing operation with rtl8169_start_xmit:
     if (tp->dirty_tx != dirty_tx) {
         tp->dirty_tx = dirty_tx;
         smp_mb();
         ...
     }
The function rtl8169_start_xmit reads tp->dirty_tx in TX_FRAGS_READY_FOR:
     if (unlikely(!TX_FRAGS_READY_FOR(tp, skb_shinfo(skb)->nr_frags))) {
         netif_err(tp, drv, dev, "BUG! Tx Ring full when queue awake!\n");
         goto err_stop_0;
     }
But there is no memory barrier around this code.

Is there a possible data race here?
If not, how this data race is avoided?


Thanks,
Jia-Ju Bai

On Thu, Jan 18, 2018 at 10:06:17PM +0800, Jia-Ju Bai wrote:

In the rt8169 driver, the function "rtl_tx" uses "smp_mb" to sync the
writing operation with rtl8169_start_xmit:
    if (tp->dirty_tx != dirty_tx) {
        tp->dirty_tx = dirty_tx;
        smp_mb();
        ...
    }
The function rtl8169_start_xmit reads tp->dirty_tx in TX_FRAGS_READY_FOR:
    if (unlikely(!TX_FRAGS_READY_FOR(tp, skb_shinfo(skb)->nr_frags))) {
        netif_err(tp, drv, dev, "BUG! Tx Ring full when queue awake!\n");
        goto err_stop_0;
    }
But there is no memory barrier around this code.

Is there a possible data race here?
If not, how this data race is avoided?

There is only 1 variable afaict. Memory barriers need at least 2 in
order to be able to do _anything_.

Peter Zijlstra [off-list ref] :
[...]

There is only 1 variable afaict. Memory barriers need at least 2 in
order to be able to do _anything_.

I don't get your point: why don't {cur_tx, dirty_tx} qualify as said
two variables ?

-- 
Ueimor

Jia-Ju Bai [off-list ref] :
[...]

The function rtl8169_start_xmit reads tp->dirty_tx in TX_FRAGS_READY_FOR:
    if (unlikely(!TX_FRAGS_READY_FOR(tp, skb_shinfo(skb)->nr_frags))) {
        netif_err(tp, drv, dev, "BUG! Tx Ring full when queue awake!\n");
        goto err_stop_0;
    }
But there is no memory barrier around this code.

Is there a possible data race here?

This code would not even be needed if rtl8169_start_xmit was only your
usual ndo_start_xmit handler: Realtek {ab / re}used it for GSO handling
(see r8169_csum_workaround).

If the test is not a no-op in this GSO context, it's racy.

-- 
Ueimor

On 2018/1/19 9:11, Francois Romieu wrote:

Jia-Ju Bai [off-list ref] :
[...]

quoted

The function rtl8169_start_xmit reads tp->dirty_tx in TX_FRAGS_READY_FOR:
     if (unlikely(!TX_FRAGS_READY_FOR(tp, skb_shinfo(skb)->nr_frags))) {
         netif_err(tp, drv, dev, "BUG! Tx Ring full when queue awake!\n");
         goto err_stop_0;
     }
But there is no memory barrier around this code.

Is there a possible data race here?

This code would not even be needed if rtl8169_start_xmit was only your
usual ndo_start_xmit handler: Realtek {ab / re}used it for GSO handling
(see r8169_csum_workaround).

If the test is not a no-op in this GSO context, it's racy.

Thanks for reply.
I didn't clearly understand your meaning...
I wonder whether there is a possible data race and whether a "smp_mb" is 
needed before this code?
By the way, do you mean that this code can be removed?


Thanks,
Jia-Ju Bai

On Fri, Jan 19, 2018 at 02:11:18AM +0100, Francois Romieu wrote:

Peter Zijlstra [off-list ref] :
[...]

quoted

There is only 1 variable afaict. Memory barriers need at least 2 in
order to be able to do _anything_.

I don't get your point: why don't {cur_tx, dirty_tx} qualify as said
two variables ?

There wasn't any cur_tx in the code you provided.

Jia-Ju Bai [off-list ref] :

On 2018/1/19 9:11, Francois Romieu wrote:

quoted

Jia-Ju Bai [off-list ref] :
[...]

quoted

The function rtl8169_start_xmit reads tp->dirty_tx in TX_FRAGS_READY_FOR:
     if (unlikely(!TX_FRAGS_READY_FOR(tp, skb_shinfo(skb)->nr_frags))) {
         netif_err(tp, drv, dev, "BUG! Tx Ring full when queue awake!\n");
         goto err_stop_0;
     }
But there is no memory barrier around this code.

Is there a possible data race here?

This code would not even be needed if rtl8169_start_xmit was only your
usual ndo_start_xmit handler: Realtek {ab / re}used it for GSO handling
(see r8169_csum_workaround).

If the test is not a no-op in this GSO context, it's racy.

Thanks for reply.
I didn't clearly understand your meaning...

It's fine.

I wonder whether there is a possible data race and whether a "smp_mb" is
needed before this code?
By the way, do you mean that this code can be removed?

This code may be removed in a driver that properly stops itself its
tx queueing in the ndo_start_xmit handler (I would still keep it as
a bug detection helper but it's just a matter of taste). That's what
the r8169 driver used to aim at.

However, since e974604b453e87f8d864371786375d3d511fdf56, there is a piece
of code where the r8169 driver iteratively uses its own ndo_start_xmit
(without even checking its return value) in r8169_csum_workaround.
It is racy. Now, let's forget races for a few seconds: how is
r8169_csum_workaround supposed to work at all given that it does not care
if (the "unlikely(...)" test in) rtl8169_start_xmit succeeds or not ?

rtl8169_start_xmit can leave the skb as-is or map it to hardware descriptors
(whence late release in rtl_tx). net/core/dev.c::dev_hard_start_xmit cares.
r8169_csum_workaround doesn't.

-- 
Ueimor

Peter Zijlstra [off-list ref] :

On Fri, Jan 19, 2018 at 02:11:18AM +0100, Francois Romieu wrote:

quoted

Peter Zijlstra [off-list ref] :
[...]

quoted

There is only 1 variable afaict. Memory barriers need at least 2 in
order to be able to do _anything_.

I don't get your point: why don't {cur_tx, dirty_tx} qualify as said
two variables ?

There wasn't any cur_tx in the code you provided.

/* A skbuff with nr_frags needs nr_frags+1 entries in the tx queue */
#define TX_FRAGS_READY_FOR(tp,nr_frags) \
        (TX_SLOTS_AVAIL(tp) >= (nr_frags + 1))

#define TX_SLOTS_AVAIL(tp) \
        (tp->dirty_tx + NUM_TX_DESC - tp->cur_tx)

Both are also used in rtl_tx.

I don't get your point. Even a single variable is scattered through
the system.

-- 
Ueimor