Thanks for the clarification on the options. I'd like to propose going
with a refined version of the v5 approach (per-instance hlist_head
with auto-resize). Approach is to submit a v6 series with 2 commits:
1. Refactor instance_create() locking to enable GFP_KERNEL_ACCOUNT
instead of GFP_ATOMIC
2. Add per-instance hash table with auto-resize (no uapi changes)

Rationale for per-instance approach over shared rhashtable:
1. Resize algorithm matched to nfqueue behavior: outstanding packet
count depends on verdict/traffic rate. rhashtable resizes based on
element count, which for nfqueue means resize-up during bursts
followed by resize-down as verdicts drain the queue to zero. This
burst-drain pattern would cause repeated resize operations. The resize
approach can be tailored to nfqueue use case to reduce resize
thrashing.
2. Per-queue memory attribution/limits:  With GFP_KERNEL_ACCOUNT, hash
table allocations are charged to the cgroup that triggered the resize,
so memory consumption is bounded by cgroup limits rather than
requiring an additional kernel-internal limit.
3. Simpler key structure: Avoids composite keys (net, queue_num,
packet_id) needed for a shared hash table

I'm open to reconsidering the shared rhashtable approach if you feel
the benefits outweigh these tradeoffs.