[BUG] act_ife: sleeping functions called in atomic context

tcf_ife_init() contains a big chunk of code executed with
ife->tcf_lock spinlock held. But that code contains several calls
to sleeping functions:
  populate_metalist() and use_all_metadata()
    -> add_metainfo()
      -> find_ife_oplist(metaid)
        -> read_lock()
        -> try_module_get(o->owner)
      -> kzalloc(sizeof(*mi), GFP_KERNEL);
      -> ops->alloc(mi, metaval);
      -> module_put(ops->owner);
  _tcf_ife_cleanup()
    -> module_put()

The same problem is actual for tcf_ife_cleanup() as well.

Found by Linux Driver Verification project (linuxtesting.org).

--
Alexey Khoroshilov
Linux Verification Center, ISPRAS
web: http://linuxtesting.org

On Thu, Jun 16, 2016 at 1:50 PM, Alexey Khoroshilov
[off-list ref] wrote:

tcf_ife_init() contains a big chunk of code executed with
ife->tcf_lock spinlock held. But that code contains several calls
to sleeping functions:
  populate_metalist() and use_all_metadata()
    -> add_metainfo()
      -> find_ife_oplist(metaid)
        -> read_lock()
        -> try_module_get(o->owner)
      -> kzalloc(sizeof(*mi), GFP_KERNEL);

Hmm, we don't need to hold that spinlock when we create a new ife action,
since we haven't inserted it yet. We do need it when we modify an existing
one. So I am thinking if we can refactor that code to avoid spinlock
whenever possible.

      -> ops->alloc(mi, metaval);
      -> module_put(ops->owner);
  _tcf_ife_cleanup()
    -> module_put()

The same problem is actual for tcf_ife_cleanup() as well.

Huh? Both module_put() and kfree() should not sleep, right?

On 16-06-16 05:43 PM, Cong Wang wrote:

On Thu, Jun 16, 2016 at 1:50 PM, Alexey Khoroshilov
[off-list ref] wrote:

quoted

tcf_ife_init() contains a big chunk of code executed with
ife->tcf_lock spinlock held. But that code contains several calls
to sleeping functions:
   populate_metalist() and use_all_metadata()
     -> add_metainfo()
       -> find_ife_oplist(metaid)
         -> read_lock()
         -> try_module_get(o->owner)
       -> kzalloc(sizeof(*mi), GFP_KERNEL);

Hmm, we don't need to hold that spinlock when we create a new ife action,
since we haven't inserted it yet. We do need it when we modify an existing
one. So I am thinking if we can refactor that code to avoid spinlock
whenever possible.

Does attached (compile tested) patch help?

quoted

       -> ops->alloc(mi, metaval);
       -> module_put(ops->owner);
   _tcf_ife_cleanup()
     -> module_put()

The same problem is actual for tcf_ife_cleanup() as well.

Huh? Both module_put() and kfree() should not sleep, right?

I dont think there is any sleeping there ;->

cheers,
jamal

On Thu, Jun 16, 2016 at 5:38 PM, Jamal Hadi Salim [off-list ref] wrote:

On 16-06-16 05:43 PM, Cong Wang wrote:

quoted

On Thu, Jun 16, 2016 at 1:50 PM, Alexey Khoroshilov
[off-list ref] wrote:

quoted

tcf_ife_init() contains a big chunk of code executed with
ife->tcf_lock spinlock held. But that code contains several calls
to sleeping functions:
   populate_metalist() and use_all_metadata()
     -> add_metainfo()
       -> find_ife_oplist(metaid)
         -> read_lock()
         -> try_module_get(o->owner)
       -> kzalloc(sizeof(*mi), GFP_KERNEL);

Hmm, we don't need to hold that spinlock when we create a new ife action,
since we haven't inserted it yet. We do need it when we modify an existing
one. So I am thinking if we can refactor that code to avoid spinlock
whenever possible.

Does attached (compile tested) patch help?

You at least miss the unlock in load_metaops_and_vet()?

I think we can just remove that tcf_lock, I am testing a patch now.

On Thu, Jun 16, 2016 at 7:14 PM, Cong Wang [off-list ref] wrote:

I think we can just remove that tcf_lock, I am testing a patch now.

Please try the attached patch, I will do more tests tomorrow.

Thanks!

On 17.06.2016 08:38, Cong Wang wrote:

On Thu, Jun 16, 2016 at 7:14 PM, Cong Wang [off-list ref] wrote:

quoted

I think we can just remove that tcf_lock, I am testing a patch now.

Please try the attached patch, I will do more tests tomorrow.

Thanks!

Looks good with two notes:
1. add_metainfo() still contains
ret = ops->alloc(mi, metaval);
that allocates memory with GFP_KERNEL.
So, I would add gfpflag argument to alloc() operation.

2. It makes sense to mention ife_mod_lock in the comment before
add_metainfo(), because ife_mod_lock is the reason to use GFP_ATOMIC there.

--
Alexey

On 16-06-17 01:38 AM, Cong Wang wrote:

On Thu, Jun 16, 2016 at 7:14 PM, Cong Wang [off-list ref] wrote:

quoted

I think we can just remove that tcf_lock, I am testing a patch now.

Please try the attached patch, I will do more tests tomorrow.

Thanks!

Cong, What tree are you using? I dont see the time aggregation patches
that I sent (and Dave took in) in your changes.

Comments:
Is GFP_ATOMIC really necessary? Thats user->kernel interface. GFP_KERNEL 
should be sufficient.
Also, it would be nice to kill the lock - but this feels like two
patches in one. 1) to fix the alloc not to be under the lock 2) to
kill said lock. Maybe split it as such for easier review.
I am using this action extensively so will be happy to test.
I think my patch is a good beginning to #1 - if you fix the forgotten
unlock and ensure we lock around updating ife fields when it exists
already (you said it in your earlier email and I thought about
that afterwards).

cheers,
jamal

On Fri, Jun 17, 2016 at 4:05 AM, Alexey Khoroshilov
[off-list ref] wrote:

On 17.06.2016 08:38, Cong Wang wrote:

quoted

On Thu, Jun 16, 2016 at 7:14 PM, Cong Wang [off-list ref] wrote:

quoted

I think we can just remove that tcf_lock, I am testing a patch now.

Please try the attached patch, I will do more tests tomorrow.

Thanks!

Looks good with two notes:
1. add_metainfo() still contains
ret = ops->alloc(mi, metaval);
that allocates memory with GFP_KERNEL.
So, I would add gfpflag argument to alloc() operation.

I thought about this too, but we just allocate 32+ bytes here,
not sure if it is really worth to pass a gfp flag.

2. It makes sense to mention ife_mod_lock in the comment before
add_metainfo(), because ife_mod_lock is the reason to use GFP_ATOMIC there.

Don't worry, it is in a separated patch, I will explain this
in the changelog. (I sent a combined patch just for review/tests.)

Thanks!

On Fri, Jun 17, 2016 at 4:07 AM, Jamal Hadi Salim [off-list ref] wrote:

On 16-06-17 01:38 AM, Cong Wang wrote:

quoted

On Thu, Jun 16, 2016 at 7:14 PM, Cong Wang [off-list ref]
wrote:

quoted

I think we can just remove that tcf_lock, I am testing a patch now.

Please try the attached patch, I will do more tests tomorrow.

Thanks!

Cong, What tree are you using? I dont see the time aggregation patches
that I sent (and Dave took in) in your changes.

My patch is against -net. (I see you already figured out your patch is
missing in -net-next.)

Or are you suggesting to rebase it for -net-next? I think it fixes some real
bug so -net is better, although it is slightly large as a bug fix.

Comments:
Is GFP_ATOMIC really necessary? Thats user->kernel interface. GFP_KERNEL
should be sufficient.

I added a read_lock(ife_mod_lock), this is why we need
GFP_ATOMIC.

Again, don't worry, this change should be in a separated patch,
you will not miss it again when I send them formally. ;)

Also, it would be nice to kill the lock - but this feels like two
patches in one. 1) to fix the alloc not to be under the lock 2) to
kill said lock. Maybe split it as such for easier review.
I am using this action extensively so will be happy to test.
I think my patch is a good beginning to #1 - if you fix the forgotten
unlock and ensure we lock around updating ife fields when it exists
already (you said it in your earlier email and I thought about
that afterwards).

Yes, it makes sense too. Your patch is smaller, if you plan to
backport it to stable, we can use your patch for -net  and -stable
and I am happy to rebase mine for -net-next.

I am fine with either way.

Thanks!

On 16-06-17 01:31 PM, Cong Wang wrote:

My patch is against -net. (I see you already figured out your patch is
missing in -net-next.)

Ok, should have re-read this email before working on the patch;->

Or are you suggesting to rebase it for -net-next? I think it fixes some real
bug so -net is better, although it is slightly large as a bug fix.

I am conflicted. There are a lot of changes in net-next at the moment;
adding this to -net seems like will definetely cause merge issues for
Dave.

Ok, Cong - patch attached and tested. Let me know what you think.

cheers,
jamal