Re: [PATCH 15/15] powerpc/powernv/sriov: Make single PE mode a per-BAR setting

On 22/07/2020 15:39, Oliver O'Halloran wrote:

On Wed, Jul 15, 2020 at 6:00 PM Alexey Kardashevskiy [off-list ref] wrote:

quoted

quoted

quoted

quoted

               *
-              * Generally, one M64 BAR maps one IOV BAR. To avoid conflict
-              * with other devices, IOV BAR size is expanded to be
-              * (total_pe * VF_BAR_size).  When VF_BAR_size is half of M64
-              * segment size , the expanded size would equal to half of the
-              * whole M64 space size, which will exhaust the M64 Space and
-              * limit the system flexibility.  This is a design decision to
-              * set the boundary to quarter of the M64 segment size.
+              * The 1/4 limit is arbitrary and can be tweaked.
               */
-             if (total_vf_bar_sz > gate) {
-                     mul = roundup_pow_of_two(total_vfs);
-                     dev_info(&pdev->dev,
-                             "VF BAR Total IOV size %llx > %llx, roundup to %d VFs\n",
-                             total_vf_bar_sz, gate, mul);
-                     iov->m64_single_mode = true;
-                     break;
-             }
-     }
+             if (vf_bar_sz > (phb->ioda.m64_segsize >> 2)) {
+                     /*
+                      * On PHB3, the minimum size alignment of M64 BAR in
+                      * single mode is 32MB. If this VF BAR is smaller than
+                      * 32MB, but still too large for a segmented window
+                      * then we can't map it and need to disable SR-IOV for
+                      * this device.

Why not use single PE mode for such BAR? Better than nothing.

Suppose you could, but I figured VFs were mainly interesting since you
could give each VF to a separate guest. If there's multiple VFs under
the same single PE BAR then they'd have to be assigned to the same

True. But with one PE per VF we can still have 15 (or 14?) isolated VFs
which is not hundreds but better than 0.

We can only use single PE BARs if the per-VF size is >= 32MB due to
the alignment requirements on P8. If the per-VF size is smaller then
we're stuck with multiple VFs inside the same BAR which is bad due to
the PAPR requirements mentioned below. Sure we could look at doing
something else, but considering this matches the current behaviour
it's a bit hard to care...

quoted

quoted

guest in order to retain the freeze/unfreeze behaviour that PAPR
requires. I guess that's how it used to work, but it seems better just
to disable them rather than having VFs which sort of work.

Well, realistically the segment size should be 8MB to make this matter
(or the whole window 2GB) which does not seem to happen so it does not
matter.

I'm not sure what you mean.

I mean how can we possibly hit this case, what m64_segsize would the
platform have to trigger this. The whole check seems useless but whatever.



-- 
Alexey