On 8/27/25 1:10 AM, John Ousterhout wrote:

On Tue, Aug 26, 2025 at 2:06 AM Paolo Abeni [off-list ref] wrote:

quoted

On 8/18/25 10:55 PM, John Ousterhout wrote:

quoted

+/**
+ * struct homa_net - Contains Homa information that is specific to a
+ * particular network namespace.
+ */
+struct homa_net {
+     /** @net: Network namespace corresponding to this structure. */
+     struct net *net;
+
+     /** @homa: Global Homa information. */
+     struct homa *homa;

It's not clear why the above 2 fields are needed. You could access
directly the global struct homa instance, and 'struct net' is usually
available when struct home_net is avail.

I have eliminated net but would like to retain homa. I have tried very
hard to avoid global variables in Homa, both for general pedagogical
reasons and because it simplifies unit testing. Right now there is no
need for a global homa except a couple of places in homa_plumbing.c,
and I'd like to maintain that encapsulation.

Note that there is no kernel convention against global per protocol
variables, when that does not prevent scaling.

quoted

quoted

+/**
+ * homa_clock() - Return a fine-grain clock value that is monotonic and
+ * consistent across cores.
+ * Return: see above.
+ */
+static inline u64 homa_clock(void)
+{
+     /* As of May 2025 there does not appear to be a portable API that
+      * meets Homa's needs:
+      * - The Intel X86 TSC works well but is not portable.
+      * - sched_clock() does not guarantee monotonicity or consistency.
+      * - ktime_get_mono_fast_ns and ktime_get_raw_fast_ns are very slow
+      *   (27 ns to read, vs 8 ns for TSC)
+      * Thus we use a hybrid approach that uses TSC (via get_cycles) where
+      * available (which should be just about everywhere Homa runs).
+      */
+#ifdef CONFIG_X86_TSC
+     return get_cycles();
+#else
+     return ktime_get_mono_fast_ns();
+#endif /* CONFIG_X86_TSC */
+}

ktime_get*() variant are fast enough to allow e.g. pktgen deals with
millions of packets x seconds. Both tsc() and ktime_get_mono_fast_ns()
suffer of various inconsistencies which will cause the most unexpected
issues in the most dangerous situation. I strongly advice against this
early optimization.

Which ktime_get variant do you recommend instead of ktime_get_mono_fast_ns?

I feel pretty strongly about retaining the use of TSC on Intel
platforms. As I have said before, Homa is attempting to operate in a
much more aggressive latency domain than Linux is used to, and
nanoseconds matter. I have been using TSC on Intel and AMD platforms
for more than 15 years and I have never had any problems. Is there a
specific inconsistency you know of that will cause "unexpected issues
in the most dangerous situations"? 

The TSC raw value depends on the current CPU. According to the relevant
documentation ktime_get_mono_fast_ns() is allowed to jump under certain
conditions: with either of them you can get sudden/unexpected tick
increases.

If not, I would prefer to retain
the use of TSC until someone can identify a real problem. Note that
the choice of clock is now well encapsulated, so if a change should
become necessary it will be very easy to make.

AFAICS, in the current revision there are several points that could
cause much greater latency - i.e. the long loops under BH lock with no
reschedule. I'm surprised they don't show as ms-latency bottle-necks
under stress test.

I suggest removing such issues before doing micro optimization that at
very least use APIs that are explicitly discouraged.

/P