Re: GTE - The hardware timestamping engine

On Sat, Mar 20, 2021 at 12:56 PM Linus Walleij [off-list ref] wrote:

Hi Dipen,

thanks for your mail!

I involved some other kernel people to get some discussion.
I think Kent Gibson can be of great help because he is using
GPIOs with high precision.

We actually discussed this a bit when adding support for
realtime timestamps.

Adding Richard Cochran as well, for drivers/ptp/, he may be able to
identify whether this should be integrated into that framework in some
form.

fullquote below

On Wed, Mar 17, 2021 at 11:29 PM Dipen Patel [off-list ref] wrote:

quoted

Nvidia Tegra SoCs have generic timestamping engine (GTE) hardware module which
can monitor SoC signals like IRQ lines and GPIO lines for state change, upon
detecting the change, it can timestamp and store in its internal hardware FIFO.
The advantage of the GTE module can be realized in applications like robotics
or autonomous vehicle where it can help record events with precise timestamp.

That sounds very useful.

Certainly the kernel shall be able to handle this.

quoted

============
For GPIO:
============
1.  GPIO has to be configured as input and IRQ must be enabled.
2.  Ask GPIO controller driver to set corresponding timestamp bit in the
    specified GPIO config register.
3.  Translate GPIO specified by the client to its internal bitmap.
3.a For example, If client specifies GPIO line 31, it could be bit 13 of GTE
    register.
4.  Set internal bits to enable monitoring in GTE module
5.  Additionally GTE driver can open up lanes for the user space application
    as a client and can send timestamping events directly to the application.

I have some concerns:

1. GPIO should for all professional applications be used with the character
device /dev/gpiochipN, under no circumstances shall the old sysfs
ABI be used for this. In this case it is necessary because the
character device provides events in a FIFO to userspace, which is
what we need.

The timestamp provided to userspace is an opaque 64bit
unsigned value. I suppose we assume it is monotonic but
you can actually augment the semantics for your specific
stamp, as long as 64 bits is gonna work.

2. The timestamp for the chardev is currently obtained in
drivers/gpio/gpiolib-cdev.c like this:

static u64 line_event_timestamp(struct line *line)
{
        if (test_bit(FLAG_EVENT_CLOCK_REALTIME, &line->desc->flags))
                return ktime_get_real_ns();

        return ktime_get_ns();
}

What you want to do is to add a new flag for hardware timestamps
and use that if available. FLAG_EVENT_CLOCK_HARDWARE?
FLAG_EVENT_CLOCK_NATIVE?

Then you need to figure out a mechanism so we can obtain
the right timestamp from the hardware event right here,
you can hook into the GPIO driver if need be, we can
figure out the gpio_chip for a certain line for sure.

So you first need to augment the userspace
ABI and the character device code to add this. See
commit 26d060e47e25f2c715a1b2c48fea391f67907a30
"gpiolib: cdev: allow edge event timestamps to be configured as REALTIME"
by Kent Gibson to see what needs to be done.

3. Also patch tools/gpio/gpio-event-mon.c to support this flag and use that
for prototyping and proof of concept.

quoted

============
For IRQ:
============

Marc Zyngier and/or Thomas Gleixner know this stuff.

It does make sense to add some infrastructure so that GPIO events
and IRQs can use the same timestamping hardware.

And certainly you will also want to use this timestamp for
IIO devices? If it is just GPIOs and IRQs today, it will be
gyroscopes and accelerometers tomorrow, am I right?

Yours,
Linus Walleij

On Sat, Mar 20, 2021 at 01:44:20PM +0100, Arnd Bergmann wrote:

Adding Richard Cochran as well, for drivers/ptp/, he may be able to
identify whether this should be integrated into that framework in some
form.

I'm not familiar with the GTE, but it sounds like it is a (free
running?) clock with time stamping inputs.  If so, then it could
expose a PHC.  That gets you functionality:

- clock_gettime() and friends
- comparison ioctl between GTE clock and CLOCK_REALTIME
- time stamping channels with programmable input selection

The mentioned applications (robotics and autonomous vehicle, so near
and dear to my heart) surely already use the PHC API for dealing with
network and system time sources, and so exposing the GTE as a PHC
means that user space programs will have a consistent API.

[ The only drawback I can see is the naming of the C language
  identifiers in include/uapi/linux/ptp_clock.h.  If that bothers
  people, then these can be changed to something more generic while
  keeping compatibility aliases. ]

Thanks,
Richard

Hi Richard,

Thanks for your input and time. Please see below follow up.

On 3/20/21 8:38 AM, Richard Cochran wrote:

On Sat, Mar 20, 2021 at 01:44:20PM +0100, Arnd Bergmann wrote:

quoted

Adding Richard Cochran as well, for drivers/ptp/, he may be able to
identify whether this should be integrated into that framework in some
form.

I'm not familiar with the GTE, but it sounds like it is a (free
running?) clock with time stamping inputs.  If so, then it could
expose a PHC.  That gets you functionality:

- clock_gettime() and friends
- comparison ioctl between GTE clock and CLOCK_REALTIME
- time stamping channels with programmable input selection

GTE gets or rather records the timestamps from the TSC
(timestamp system coutner) so its not attached to GTE as any
one can access TSC, so not sure if we really need to implement PHC
and/or clock_* and friends for the GTE. I believe burden to find correlation
between various clock domains should be on the clients, consider below
example.

Networking client has access to both PTP and GTE, it would be its job
to find the correlations if that is at all needed based on whatever
use case that client serves. GTE in above may come in picture if said client
has some GPIO configured and wants timestamp on it.

Sorry if I misunderstood anything, you can elaborate more as I am also
interested in how GTE can fit in PTP framework and which usecase it can
help doing so.

The mentioned applications (robotics and autonomous vehicle, so near
and dear to my heart) surely already use the PHC API for dealing with
network and system time sources, and so exposing the GTE as a PHC
means that user space programs will have a consistent API.

[ The only drawback I can see is the naming of the C language
  identifiers in include/uapi/linux/ptp_clock.h.  If that bothers
  people, then these can be changed to something more generic while
  keeping compatibility aliases. ]

Thanks,
Richard

Thanks,
Best Regards,
Dipen Patel

On Mon, Mar 22, 2021 at 01:33:38PM -0700, Dipen Patel wrote:

Hi Richard,

Thanks for your input and time. Please see below follow up.

On 3/20/21 8:38 AM, Richard Cochran wrote:

quoted

On Sat, Mar 20, 2021 at 01:44:20PM +0100, Arnd Bergmann wrote:

quoted

Adding Richard Cochran as well, for drivers/ptp/, he may be able to
identify whether this should be integrated into that framework in some
form.

I'm not familiar with the GTE, but it sounds like it is a (free
running?) clock with time stamping inputs.  If so, then it could
expose a PHC.  That gets you functionality:

- clock_gettime() and friends
- comparison ioctl between GTE clock and CLOCK_REALTIME
- time stamping channels with programmable input selection

GTE gets or rather records the timestamps from the TSC
(timestamp system coutner) so its not attached to GTE as any
one can access TSC, so not sure if we really need to implement PHC
and/or clock_* and friends for the GTE. I believe burden to find correlation
between various clock domains should be on the clients, consider below
example.

I agree. My understanding is the the TSC is basically an SoC-wide clock
that can be (and is) used by several hardware blocks. There's an
interface for software to read out the value, but it's part of a block
called TKE (time-keeping engine, if I recall correctly) that implements
various clock sources and watchdog functionality.

As a matter of fact, I recall typing up a driver for that at some point
but I don't recall if I ever sent it out or what became of it. I can't
find it upstream at least.

Anyway, I think given that the GTE doesn't provide that clock itself but
rather just a means of taking a snapshot of that clock and stamping
certain events with that, it makes more sense to provide that clock from
the TKE driver.

Thierry

On Tue, Mar 23, 2021 at 10:03:18AM +0100, Thierry Reding wrote:

I agree. My understanding is the the TSC is basically an SoC-wide clock
that can be (and is) used by several hardware blocks. There's an
interface for software to read out the value, but it's part of a block
called TKE (time-keeping engine, if I recall correctly) that implements
various clock sources and watchdog functionality.

...

Anyway, I think given that the GTE doesn't provide that clock itself but
rather just a means of taking a snapshot of that clock and stamping
certain events with that, it makes more sense to provide that clock from
the TKE driver.

It sounds like TKE + GTE together act like a PHC, and GTE doesn't
need/want its own SW interface.

Thanks,
Richard