RE: [net-next 5/6] ixgbe: add hardware timestamping support

-----Original Message-----
From: Richard Cochran [mailto:richardcochran@gmail.com]
Sent: Thursday, October 20, 2011 7:57 AM
To: Jacob Keller
Cc: Kirsher, Jeffrey T; davem@davemloft.net; Keller, Jacob E;
netdev@vger.kernel.org; gospo@redhat.com; sassmann@redhat.com
Subject: Re: [net-next 5/6] ixgbe: add hardware timestamping support

On Wed, Oct 19, 2011 at 10:04:33AM -0700, Jacob Keller wrote:

quoted

On Mon, Oct 17, 2011 at 9:44 AM, Richard Cochran

[off-list ref] wrote:

quoted

quoted

So, is this wrap around due to the fact that you are tied to the
system time via time_compare? Or, putting it another way, can't you
program the hardware time stamping unit so that the registers have
some reasonable resolution (like 64 bits worth of nanoseconds) and
just offer RAW timestamps?

The wrap around is due to hardware limitations. The ixgbe devices
cannot support 64bits worth of nanoseconds and still have the ability
to adjust the frequency in parts per billion. A larger increment
increases the resolution available for frequency adjustments, but
decreases the time it takes for the cycle counter to wrap around.

Oh, well. That stinks.

I think you do want to offer ppb adjustment.

Correct, which is why the cycle counter wraps around every 35 seconds.

quoted

quoted

I would really like to move away from the timecompare hacks and
towards a proper PHC->SYS PPS solution.

I agree that this is the correct approach. The timecompare
functionality does have issues.

And these cards are highlighting timecompare weaknesses I had not even
thought of.

I expect that if you offer the RAW time stamps, then it should be
possible to have the time stamp values always correct (or nearly so)
even with a changing link speed. If the link speed change gives an
interrupt, then the ISR can reprogram the frequency compensation
registers and let the counter continue.

The cyclecounter is based off of the DMA clock on the NIC which changes frequency with the link speed. So at 10G link, the DMA ticks once every 6.4ns. The cycle counter gets a value (specified in the TIMINCA register) added to it every DMA tick. In order to allow for ppb adjustments to the cycle counter, I have the TIMINCA value be as many bits wide as possible. Then I use the cyclecounter/timecounter structures to detect wraparound and convert to a ns value.

If we return the raw cycle counter stamps directly, they would not be measured in nanoseconds, but in a division of the DMA clock tick (DMA clock tick / TIMINCA value). This means for the values I chose we are somewhere in the range of femto seconds or so. The problem is that all of the upper stack expects values as nanoseconds. We wouldn't be able to frequency adjust in ppb to get down to nanoseconds again.

This is due to a limitation in the way the hardware was designed, (Ideally it would allow for precise adjustments, but still provide a nanosecond counter. The 82580 igb device does this.)

quoted

quoted

Again, doing the update thing on every packet won't work for real
world PTP scenarios.

Which is why the PHC solution is better. Work on implementing this
support is in progress. Out of curiosity, what is the sync rate for
the scenario that breaks this? I would like to try that rate out on

my

quoted

setup.

For the audio/video profile, they have a max of 32 sync packets per
second. Not sure about delay request rate, maybe 16 per second.

Thanks :)

Thanks,
Richard