[PATCH v9 09/14] iio: afe: rescale: fix accuracy for small fractional scales
From: Liam Beguin <hidden>
Date: 2021-11-15 03:45:53
Also in:
linux-iio, lkml
Subsystem:
iio subsystem and drivers, iio unit converter, the rest · Maintainers:
Jonathan Cameron, Peter Rosin, Linus Torvalds
From: Liam Beguin <redacted> The approximation caused by integer divisions can be costly on smaller scale values since the decimal part is significant compared to the integer part. Switch to an IIO_VAL_INT_PLUS_NANO scale type in such cases to maintain accuracy. Signed-off-by: Liam Beguin <redacted> --- drivers/iio/afe/iio-rescale.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)
diff --git a/drivers/iio/afe/iio-rescale.c b/drivers/iio/afe/iio-rescale.c
index 17036130d364..8a2f1c0ca5a3 100644
--- a/drivers/iio/afe/iio-rescale.c
+++ b/drivers/iio/afe/iio-rescale.c@@ -22,7 +22,7 @@ int rescale_process_scale(struct rescale *rescale, int scale_type, int *val, int *val2) { s64 tmp; - s32 rem; + s32 rem, rem2; u32 mult; u32 neg;
@@ -38,9 +38,26 @@ int rescale_process_scale(struct rescale *rescale, int scale_type, tmp = (s64)*val * 1000000000LL; tmp = div_s64(tmp, rescale->denominator); tmp *= rescale->numerator; - tmp = div_s64(tmp, 1000000000LL); + + tmp = div_s64_rem(tmp, 1000000000LL, &rem); *val = tmp; - return scale_type; + + if (!rem) + return scale_type; + + if (scale_type == IIO_VAL_FRACTIONAL) + tmp = *val2; + else + tmp = 1 << *val2; + + rem2 = *val % (int)tmp; + *val = *val / (int)tmp; + + *val2 = rem / (int)tmp; + if (rem2) + *val2 += div_s64((s64)rem2 * 1000000000LL, tmp); + + return IIO_VAL_INT_PLUS_NANO; case IIO_VAL_INT_PLUS_NANO: case IIO_VAL_INT_PLUS_MICRO: if (scale_type == IIO_VAL_INT_PLUS_NANO)
--
2.32.0.452.g940fe202adcb