Hi,

I just quickly tried following the flow_offload_teardown() path instead of clearing IPS_OFFLOAD in flow_offload_del() and it does have some side effects. The flow is added again before the HW has actually reported it to be NF_FLOW_HW_DEAD. 

The sequence with my patch is:
  : Retire -> Remove from hw tables -> Remove from sw tables -> kfree(flow) -> flow_offload_add()

But if flow_offload_teardown() is called on expire I see:
  : Retire -> Remove from hw tables -> flow_offload_add() -> Remove from sw tables -> kfree(flow)
 
I need to investigate why this happens, maybe the IPS_OFFLOAD flag is cleared too early and should not be cleared until the flow is actually removed, like I do? Maybe the issue is not seen before because on timeout or flow_is_dying() no packet arrive to create the flow again prematurely?

Thanks,
Michael