^
ffff88803d750480: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff88803d750500: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
==================================================================
---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.
syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
syzbot can test patches for this issue, for details see:
https://goo.gl/tpsmEJ#testing-patches
I think, there a missing lock in this function:
for_each_possible_cpu(cpu) {
ct = per_cpu(nft_ct_pcpu_template, cpu);
if (!ct)
break;
nf_ct_put(ct);
per_cpu(nft_ct_pcpu_template, cpu) = NULL;
}
Syzbot hit a UAF in nft_ct_tmpl_put_pcpu() (*), but freed template
should be NULL.
So I suspect following scenario:
CPU0: CPU1:
= per_cpu()
= per_cpu()
nf_ct_put
per_cpu = NULL
nf_ct_put()
* UAF *
#syz test
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
With regards,
Pavel Skripkin
__nft_ct_set_destroy net/netfilter/nft_ct.c:529 [inline]
__nft_ct_set_destroy net/netfilter/nft_ct.c:518 [inline]
nft_ct_set_init+0x41e/0x750 net/netfilter/nft_ct.c:614
nf_tables_newexpr net/netfilter/nf_tables_api.c:2742 [inline]
nft_expr_init+0x145/0x2d0 net/netfilter/nf_tables_api.c:2780
nft_set_elem_expr_alloc+0x27/0x280 net/netfilter/nf_tables_api.c:5284
nf_tables_newset+0x208a/0x32f0 net/netfilter/nf_tables_api.c:4389
nfnetlink_rcv_batch+0x1710/0x25f0 net/netfilter/nfnetlink.c:513
nfnetlink_rcv_skb_batch net/netfilter/nfnetlink.c:634 [inline]
nfnetlink_rcv+0x3af/0x420 net/netfilter/nfnetlink.c:652
netlink_unicast_kernel net/netlink/af_netlink.c:1314 [inline]
netlink_unicast+0x533/0x7d0 net/netlink/af_netlink.c:1340
netlink_sendmsg+0x86d/0xdb0 net/netlink/af_netlink.c:1929
sock_sendmsg_nosec net/socket.c:703 [inline]
sock_sendmsg+0xcf/0x120 net/socket.c:723
____sys_sendmsg+0x6e8/0x810 net/socket.c:2392
___sys_sendmsg+0xf3/0x170 net/socket.c:2446
__sys_sendmsg+0xe5/0x1b0 net/socket.c:2475
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x44/0xae
The buggy address belongs to the object at ffff88803d750400
which belongs to the cache kmalloc-512 of size 512
The buggy address is located 0 bytes inside of
512-byte region [ffff88803d750400, ffff88803d750600)
The buggy address belongs to the page:
page:ffffea0000f5d400 refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x3d750
head:ffffea0000f5d400 order:2 compound_mapcount:0 compound_pincount:0
flags: 0xfff00000010200(slab|head|node=0|zone=1|lastcpupid=0x7ff)
raw: 00fff00000010200 0000000000000000 dead000000000122 ffff888010841c80
raw: 0000000000000000 0000000080100010 00000001ffffffff 0000000000000000
page dumped because: kasan: bad access detected
page_owner tracks the page as allocated
page last allocated via order 2, migratetype Unmovable, gfp_mask 0xd20c0(__GFP_IO|__GFP_FS|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP|__GFP_NOMEMALLOC), pid 9789, ts 226704064982, free_ts 0
prep_new_page mm/page_alloc.c:2436 [inline]
get_page_from_freelist+0xa72/0x2f80 mm/page_alloc.c:4169
__alloc_pages+0x1b2/0x500 mm/page_alloc.c:5391
alloc_pages+0x18c/0x2a0 mm/mempolicy.c:2244
alloc_slab_page mm/slub.c:1688 [inline]
allocate_slab+0x32e/0x4b0 mm/slub.c:1828
new_slab mm/slub.c:1891 [inline]
new_slab_objects mm/slub.c:2637 [inline]
___slab_alloc+0x4ba/0x820 mm/slub.c:2800
__slab_alloc.constprop.0+0xa7/0xf0 mm/slub.c:2840
slab_alloc_node mm/slub.c:2922 [inline]
slab_alloc mm/slub.c:2964 [inline]
kmem_cache_alloc_trace+0x30f/0x3c0 mm/slub.c:2981
kmalloc include/linux/slab.h:591 [inline]
kzalloc include/linux/slab.h:721 [inline]
nf_ct_tmpl_alloc+0x8d/0x270 net/netfilter/nf_conntrack_core.c:569
nft_ct_tmpl_alloc_pcpu net/netfilter/nft_ct.c:371 [inline]
nft_ct_set_init+0x4d6/0x750 net/netfilter/nft_ct.c:567
nf_tables_newexpr net/netfilter/nf_tables_api.c:2742 [inline]
nft_expr_init+0x145/0x2d0 net/netfilter/nf_tables_api.c:2780
nft_set_elem_expr_alloc+0x27/0x280 net/netfilter/nf_tables_api.c:5284
nf_tables_newset+0x208a/0x32f0 net/netfilter/nf_tables_api.c:4389
nfnetlink_rcv_batch+0x1710/0x25f0 net/netfilter/nfnetlink.c:513
nfnetlink_rcv_skb_batch net/netfilter/nfnetlink.c:634 [inline]
nfnetlink_rcv+0x3af/0x420 net/netfilter/nfnetlink.c:652
netlink_unicast_kernel net/netlink/af_netlink.c:1314 [inline]
netlink_unicast+0x533/0x7d0 net/netlink/af_netlink.c:1340
netlink_sendmsg+0x86d/0xdb0 net/netlink/af_netlink.c:1929
page_owner free stack trace missing
Memory state around the buggy address:
ffff88803d750300: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff88803d750380: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
I think, there a missing lock in this function:
for_each_possible_cpu(cpu) {
ct = per_cpu(nft_ct_pcpu_template, cpu);
if (!ct)
break;
nf_ct_put(ct);
per_cpu(nft_ct_pcpu_template, cpu) = NULL;
}
Syzbot hit a UAF in nft_ct_tmpl_put_pcpu() (*), but freed template should be
NULL.
So I suspect following scenario:
CPU0: CPU1:
= per_cpu()
= per_cpu()
nf_ct_put
per_cpu = NULL
nf_ct_put()
* UAF *
Yes and no. The above is fine since pcpu will return different pointers
for cpu 0 and 1.
The race is between two different net namespaces that race when
changing nft_ct_pcpu_template_refcnt.
This happens since
commit f102d66b335a417d4848da9441f585695a838934
netfilter: nf_tables: use dedicated mutex to guard transactions
Before this, all transactions were serialized by a global mutex,
now we only serialize transactions in the same netns.
Its probably best to add
DEFINE_MUTEX(nft_ct_pcpu_mutex) and then acquire that when we need to
inc/dec the nft_ct_pcpu_template_refcnt so we can't have two distinct
cpus hitting a zero refcount.
Would you send a patch for this?
Thanks.
From: Pavel Skripkin <hidden> Date: 2021-08-09 21:17:06
On 8/9/21 11:39 PM, Florian Westphal wrote:
Pavel Skripkin [off-list ref] wrote:
quoted
I think, there a missing lock in this function:
for_each_possible_cpu(cpu) {
ct = per_cpu(nft_ct_pcpu_template, cpu);
(*)
quoted
if (!ct) >> break;
nf_ct_put(ct);
per_cpu(nft_ct_pcpu_template, cpu) = NULL;
}
Syzbot hit a UAF in nft_ct_tmpl_put_pcpu() (*), but freed template should be
NULL.
So I suspect following scenario:
CPU0: CPU1:
= per_cpu()
= per_cpu()
nf_ct_put
per_cpu = NULL
nf_ct_put()
* UAF *
Hi, Florian!
Yes and no. The above is fine since pcpu will return different pointers
for cpu 0 and 1.
Dumb question: why per_cpu() will return 2 different pointers for CPU 1
and CPU 0? As I understand for_each_possible_cpu() will iterate over all
CPUs which could ever be enabled. So, we can hit situation when 2
concurrent processes call per_cpu() with same cpu value (*).
The race is between two different net namespaces that race when
changing nft_ct_pcpu_template_refcnt.
This happens since
commit f102d66b335a417d4848da9441f585695a838934
netfilter: nf_tables: use dedicated mutex to guard transactions
Before this, all transactions were serialized by a global mutex,
now we only serialize transactions in the same netns.
Its probably best to add
DEFINE_MUTEX(nft_ct_pcpu_mutex) and then acquire that when we need to
inc/dec the nft_ct_pcpu_template_refcnt so we can't have two distinct
cpus hitting a zero refcount.
Would you send a patch for this?
Anyway, I think, moving locking a bit higher is good here, let's test
it. I will prepare a patch, if it will pass syzbot testing, thanks!
#syz test
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
With regards,
Pavel Skripkin
Dumb question: why per_cpu() will return 2 different pointers for CPU 1 and
CPU 0? As I understand for_each_possible_cpu() will iterate over all
CPUs which could ever be enabled. So, we can hit situation when 2 concurrent
processes call per_cpu() with same cpu value (*).
Yes, that is what I was trying to say, the race is that we can have > 1
processes here ever since the global transaction mutex was removed in 2018.
Anyway, I think, moving locking a bit higher is good here, let's test it. I
will prepare a patch, if it will pass syzbot testing, thanks!