On Thu, May 04, 2017 at 04:50:23PM +0200, Auger Eric wrote:

Hi,

On 04/05/2017 15:23, Christoffer Dall wrote:

quoted

On Thu, May 04, 2017 at 01:44:21PM +0200, Eric Auger wrote:

quoted

Add description for how to access ITS registers and how to save/restore
ITS tables into/from memory.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v5 -> v6:
- add restoration ordering
- 256B -> 256 Byte aligned
- DTE Size is number of bits for the EVENTID
- s/GITS_READR/GITS_CREADR

v4 -> v5:
- take into account Christoffer's comments
- pending table save on GICV3 side now

v3 -> v4:
- take into account Peter's comments:
  - typos
  - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
  - add a validity bit in DTE
  - document all fields in CTE and ITE
  - document ABI revision
- take into account Andre's comments:
  - document restrictions about GITS_CREADR writing and GITS_IIDR
  - document -EBUSY error if one or more VCPUS are runnning
  - document 64b registers only can be accessed with 64b access
- itt_addr field matches bits [51:8] of the itt_addr

v1 -> v2:
- DTE and ITE now are 8 bytes
- DTE and ITE now indexed by deviceid/eventid
- use ITE name instead of ITTE
- mentions ITT_addr matches bits [51:8] of the actual address
- mentions LE layout
---
 Documentation/virtual/kvm/devices/arm-vgic-its.txt | 117 +++++++++++++++++++++
 1 file changed, 117 insertions(+)

diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
index 6081a5b..863c359 100644
--- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
+++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt

@@ -32,7 +32,124 @@ Groups:
     KVM_DEV_ARM_VGIC_CTRL_INIT
       request the initialization of the ITS, no additional parameter in
       kvm_device_attr.addr.
+
+    KVM_DEV_ARM_ITS_SAVE_TABLES
+      save the ITS table data into guest RAM, at the location provisioned
+      by the guest in corresponding registers/table entries.
+
+      The layout of the tables in guest memory defines an ABI. The entries
+      are laid out in little endian format as described in the last paragraph.
+
+    KVM_DEV_ARM_ITS_RESTORE_TABLES
+      restore the ITS tables from guest RAM to ITS internal structures.
+
+      The GICV3 must be restored before the ITS and all ITS registers but
+      the GITS_CTLR must be restored before restoring the ITS tables.
+
+      The GITS_IIDR read-only register must also be restored before
+      the table restore as the IIDR revision field encodes the ABI revision.

the table restore?  "before calling KVM_DEV_ARM_ITS_RESTORE_TABLES" ?
(there's a potential confusion between calling this ioctl and simply
restoring the memory containing the tables).

OK

quoted

quoted

+
+      The expected ordering when restoring the GICv3/ITS is described in section
+      "ITS Restoration Sequence".
+
   Errors:
     -ENXIO:  ITS not properly configured as required prior to setting
              this attribute
     -ENOMEM: Memory shortage when allocating ITS internal data
+    -EINVAL: Inconsistent restored data
+    -EFAULT: Invalid guest ram access
+    -EBUSY:  One or more VCPUS are running
+
+  KVM_DEV_ARM_VGIC_GRP_ITS_REGS
+  Attributes:
+      The attr field of kvm_device_attr encodes the offset of the
+      ITS register, relative to the ITS control frame base address
+      (ITS_base).
+
+      kvm_device_attr.addr points to a __u64 value whatever the width
+      of the addressed register (32/64 bits). 64 bit registers can only
+      be accessed with full length.
+
+      Writes to read-only registers are ignored by the kernel except for:
+      - GITS_CREADR. It needs to be restored otherwise commands in the queue
+        will be re-executed after restoring CWRITER. GITS_CREADR must be
+        restored before restoring the GITS_CTLR which is likely to enable the
+        ITS. Also it needs to be restored after GITS_CBASER since a write to
+        GITS_CBASER resets GITS_CREADR.

uber nit: s/needs to be/must be/g in all occurences above.

OK

quoted

quoted

+      - GITS_IIDR. Its Revision field encodes the table layout ABI revision.

s/Its/The/

OK. Argh, I meant its (~his) ;-)

You wrote it correctly, it's just that it can be misleading, so I'm
really in the business of beutifying the ABI text here :)

quoted

the revision of the table layout ABI.

quoted

+        In the future we might implement direct injection of virtual LPIS.

LPIs

OK

quoted

quoted

+        This will require an upgrade of the table layout and an evolution of
+        the ABI. GITS_IIDR must be restored before the table restoration.

before calling KVM_DEV_ARM_ITS_RESTORE_TABLES.

OK

quoted

quoted

+
+      For other registers, getting or setting a register has the same
+      effect as reading/writing the register on real hardware.
+  Errors:
+    -ENXIO: Offset does not correspond to any supported register
+    -EFAULT: Invalid user pointer for attr->addr
+    -EINVAL: Offset is not 64-bit aligned
+    -EBUSY: one or more VCPUS are running
+
+ ITS Restoration Sequence:

s/Restoration/restore/

OK

quoted

quoted

+ -------------------------
+
+The following ordering must be followed when restoring the GIC and the ITS:
+a) restore all guest memory and create vcpus
+b) initialize the GIC and ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)

are these really coupled or can you do everything related to the gic
first, and then everything related to the ITS afterwards?

no they aren't.

quoted

quoted

+c) provide GIC and ITS base addresses (KVM_DEV_ARM_VGIC_GRP_ADDR)

same

no they aren't

quoted

quoted

+d) restore all redistributors
+e) restore the ITS in the following order:
+   1. Restore GITS_CBASER
+   2. Restore all other GITS_ registers, except GITS_CTLR!
+   3. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
+   4. Restore GITS_CTLR
+f) start vcpus

I would omit (f) from here.

In that case I think it would make sense to describe the restore
sequence for GICV3 RDIST in GICV3 device doc and not here. This would
alleviate this very description.

restoring RDIST regs require a previous GICV3 KVM_DEV_ARM_VGIC_CTRL_INIT
and KVM_DEV_ARM_VGIC_GRP_ADDR I think.

So description here would become

"
a) restore all guest memory and create vcpus
b) restore all redistributors
c) initialize the ITS and then provide its base address    	
(KVM_DEV_ARM_VGIC_CTRL_INIT, KVM_DEV_ARM_VGIC_GRP_ADDR)
c) restore the ITS in the following order:
	1. Restore GITS_CBASER
	2. Restore all other GITS_ registers, except GITS_CTLR!
	3. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
	4. Restore GITS_CTLR

Then vcpus can be started.
"

What do you think?

Yes, looks good.

quoted

quoted

+
+ ITS Table ABI REV0:
+ -------------------
+
+ Revision 0 of the ABI only supports physical LPIs.
+
+ The device table and ITT are indexed by the deviceid and eventid,
+ respectively. The collection table is not indexed by collectionid:
+ CTE are written in the table in the order of collection creation. All

nit: CTEs

OK

quoted

quoted

+ entries are 8 bytes.
+
+ Device Table Entry (DTE):
+
+ bits:     | 63| 62 ... 49 | 48 ... 5 | 4 ... 0 |
+ values:   | V |   next    | ITT_addr |  Size   |
+
+ where;
+ - V indicates whether the entry is valid. If not, other fields
+   are not meaningful.
+ - next: equals to 0 if this entry is the last one; otherwise it
+   corresponds to the deviceid offset to the next DTE, capped by
+   2^14 -1.
+ - ITT_addr matches bits [51:8] of the ITT address (256 Byte aligned).
+ - Size specifies the supported number of bits for the eventid,
+   minus one
+
+ Collection Table Entry (CTE):
+
+ bits:     | 63| 62 ..  52  | 51 ... 16 | 15  ...   0 |
+ values:   | V |    RES0    |  RDBase   |    ICID     |
+
+ where:
+ - V indicates whether the entry is valid. If not, other fields are
+   not meaningful.
+ - RES0: reserved field with Should-Be-Zero-or-Preserved behavior.
+ - RDBase is the PE number (GICR_TYPER.Processor_Number semantic),
+ - ICID is the collection ID
+
+ Interrupt Translation Entry (ITE):
+
+ bits:     | 63 ... 48 | 47 ... 16 | 15 ... 0 |
+ values:   |    next   |   pINTID  |  ICID    |
+
+ where:
+ - next: equals to 0 if this entry is the last one; otherwise it corresponds
+   to the eventid offset to the next ITE capped by 2^16 -1.
+ - pINTID is the physical LPI ID; if zero, it means the entry is not valid
+   and other fields are not meaningful.
+ - ICID is the collection ID
+
-- 
2.5.5

The only outstanding issue here is the restore order part.  I don't wont
to specify a stronger order than absolutely necessary here.

see above

Yep, I think we're there.

Thanks,
-Christoffer