On Sat, Feb 11, 2017 at 09:58:10AM +0100, Richard Cochran wrote:

If I am not mistaken, then you can skip the cases val==2 and val==3,
because they are equivalent to val==4 and 6.

I took a stab at this, and you can see the result, below.  My version
has lower average error than yours in the interval 1 < ppb < 60000,
and it uses only 8 64-bit divisions.  Outside of that interval, your
version has lower error.

So, at the very least, you should introduce a threshold and use this
algorithm for adjustments under 60 ppm.  Better yet, find a way to use
fewer divisions for adjustments greater and 60 ppm...

Thanks,
Richard

---
#include <stdint.h>

#define TEN9		(1000000000UL)

unsigned int calc_min_integer(uint64_t ppb, uint64_t *M)
{
	uint64_t err, m, min, n, N, p2, reg;

	min = TEN9;
	for (n = 4; n <= 7; n++) {
		m = n * TEN9;
		m = (m + ppb/2) / ppb;

		/*truncate to HW resolution*/
		reg = (m + 8) / 16;
		m = reg * 16;

		p2 = (n * TEN9 + m/2) / m;

		err = ppb > p2 ? ppb - p2 : p2 - ppb;

		if (min >= err) {
			min = err;
			N = n;
			*M = m;
		}
	}
	return N;
}