[PATCHv3 00/11] crypto: omap HW crypto fixes

Hi,

This revision took quite a bit time to craft due to the rework needed
for sham buffer handling and export/import. I ended up implementing
a flush functionality for draining out the sham buffer when doing
export/import; just shrinking the buffer to sufficiently small size
impacted the performance with small data chunks too much so I dropped
this approach.

The series also fixes a couple of existing issues with omap2/omap3
hardware acceleration, I ran a full boot test / crypto manager
test suite on all boards accessible to me now.

Based on top of latest mainline, which is somewhere before 4.8-rc1
as of writing this, I am unable to rebase the series during the next
three weeks so wanted to get this out now. Targeted for 4.9 merge
window, some fixes could be picked up earlier though if needed.

-Tero

Some of the call paths of OMAP SHA driver can avoid executing the next
step of the crypto queue under tasklet; instead, execute the next step
directly via function call. This avoids a costly round-trip via the
scheduler giving a slight performance boost.

Signed-off-by: Tero Kristo <redacted>
---
 drivers/crypto/omap-sham.c | 20 +++++++++++++++-----
 1 file changed, 15 insertions(+), 5 deletions(-)

diff --git a/drivers/crypto/omap-sham.c b/drivers/crypto/omap-sham.c
index 7fe4eef..fd50005 100644
--- a/drivers/crypto/omap-sham.c
+++ b/drivers/crypto/omap-sham.c

@@ -1005,9 +1005,6 @@ static void omap_sham_finish_req(struct ahash_request *req, int err)
 
 	if (req->base.complete)
 		req->base.complete(&req->base, err);
-
-	/* handle new request */
-	tasklet_schedule(&dd->done_task);
 }
 
 static int omap_sham_handle_queue(struct omap_sham_dev *dd,

@@ -1018,6 +1015,7 @@ static int omap_sham_handle_queue(struct omap_sham_dev *dd,
 	unsigned long flags;
 	int err = 0, ret = 0;
 
+retry:
 	spin_lock_irqsave(&dd->lock, flags);
 	if (req)
 		ret = ahash_enqueue_request(&dd->queue, req);

@@ -1061,11 +1059,19 @@ static int omap_sham_handle_queue(struct omap_sham_dev *dd,
 		err = omap_sham_final_req(dd);
 	}
 err1:
-	if (err != -EINPROGRESS)
+	dev_dbg(dd->dev, "exit, err: %d\n", err);
+
+	if (err != -EINPROGRESS) {
 		/* done_task will not finish it, so do it here */
 		omap_sham_finish_req(req, err);
+		req = NULL;
 
-	dev_dbg(dd->dev, "exit, err: %d\n", err);
+		/*
+		 * Execute next request immediately if there is anything
+		 * in queue.
+		 */
+		goto retry;
+	}
 
 	return ret;
 }

@@ -1653,6 +1659,10 @@ finish:
 	dev_dbg(dd->dev, "update done: err: %d\n", err);
 	/* finish curent request */
 	omap_sham_finish_req(dd->req, err);
+
+	/* If we are not busy, process next req */
+	if (!test_bit(FLAGS_BUSY, &dd->flags))
+		omap_sham_handle_queue(dd, NULL);
 }
 
 static irqreturn_t omap_sham_irq_common(struct omap_sham_dev *dd)

-- 
1.9.1

This flushes any full blocks of data from the data buffer. Required for
implementing the export/import APIs for the driver, as the flush allows
saving a much smaller context; basically only one block of buffer is
required.

Signed-off-by: Tero Kristo <redacted>
---
 drivers/crypto/omap-sham.c | 60 ++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 55 insertions(+), 5 deletions(-)

diff --git a/drivers/crypto/omap-sham.c b/drivers/crypto/omap-sham.c
index fd50005..6e53944 100644
--- a/drivers/crypto/omap-sham.c
+++ b/drivers/crypto/omap-sham.c

@@ -112,6 +112,7 @@
 #define FLAGS_DMA_READY		6
 #define FLAGS_AUTO_XOR		7
 #define FLAGS_BE32_SHA1		8
+#define FLAGS_FLUSH		9
 /* context flags */
 #define FLAGS_FINUP		16
 #define FLAGS_SG		17

@@ -996,15 +997,16 @@ static void omap_sham_finish_req(struct ahash_request *req, int err)
 		ctx->flags |= BIT(FLAGS_ERROR);
 	}
 
-	/* atomic operation is not needed here */
-	dd->flags &= ~(BIT(FLAGS_BUSY) | BIT(FLAGS_FINAL) | BIT(FLAGS_CPU) |
-			BIT(FLAGS_DMA_READY) | BIT(FLAGS_OUTPUT_READY));
-
 	pm_runtime_mark_last_busy(dd->dev);
 	pm_runtime_put_autosuspend(dd->dev);
 
-	if (req->base.complete)
+	if (!test_bit(FLAGS_FLUSH, &dd->flags) && req->base.complete)
 		req->base.complete(&req->base, err);
+
+	/* atomic operation is not needed here */
+	dd->flags &= ~(BIT(FLAGS_BUSY) | BIT(FLAGS_FINAL) | BIT(FLAGS_CPU) |
+		       BIT(FLAGS_DMA_READY) | BIT(FLAGS_OUTPUT_READY) |
+		       BIT(FLAGS_FLUSH));
 }
 
 static int omap_sham_handle_queue(struct omap_sham_dev *dd,

@@ -1329,6 +1331,54 @@ static void omap_sham_cra_exit(struct crypto_tfm *tfm)
 	}
 }
 
+static int omap_sham_flush(struct ahash_request *req)
+{
+	struct omap_sham_reqctx *rctx = ahash_request_ctx(req);
+	struct omap_sham_dev *dd = rctx->dd;
+	int ret, len, bs;
+	unsigned long flags;
+
+	bs = get_block_size(rctx);
+
+	len = rctx->bufcnt / bs * bs;
+
+	spin_lock_irqsave(&dd->lock, flags);
+
+	if (test_bit(FLAGS_BUSY, &dd->flags)) {
+		ret = -EINPROGRESS;
+		goto exit_unlock;
+	}
+
+	if (!len) {
+		ret = 0;
+		goto exit_unlock;
+	}
+
+	set_bit(FLAGS_BUSY, &dd->flags);
+
+	spin_unlock_irqrestore(&dd->lock, flags);
+
+	rctx->total = 0;
+
+	ret = omap_sham_hw_init(dd);
+	if (ret)
+		goto exit_unlock;
+
+	set_bit(FLAGS_FLUSH, &dd->flags);
+
+	ret = omap_sham_xmit_cpu(dd, rctx->buffer, len, 0);
+	if (ret == -EINPROGRESS) {
+		memcpy(rctx->buffer, rctx->buffer + len, rctx->bufcnt - len);
+		rctx->bufcnt -= len;
+	}
+
+	return ret;
+
+exit_unlock:
+	spin_unlock_irqrestore(&dd->lock, flags);
+	return ret;
+}
+
 static struct ahash_alg algs_sha1_md5[] = {
 {
 	.init		= omap_sham_init,

-- 
1.9.1

Context export/import are now required for ahash algorithms due to
required support in algif_hash. Implement these for OMAP SHA driver,
saving and restoring the internal state of the driver.

Signed-off-by: Tero Kristo <redacted>
---
 drivers/crypto/omap-sham.c | 31 +++++++++++++++++++++++++++++--
 1 file changed, 29 insertions(+), 2 deletions(-)

diff --git a/drivers/crypto/omap-sham.c b/drivers/crypto/omap-sham.c
index 6e53944..aa71e61 100644
--- a/drivers/crypto/omap-sham.c
+++ b/drivers/crypto/omap-sham.c

@@ -1379,6 +1379,27 @@ exit_unlock:
 	return ret;
 }
 
+static int omap_sham_export(struct ahash_request *req, void *out)
+{
+	struct omap_sham_reqctx *rctx = ahash_request_ctx(req);
+
+	while (omap_sham_flush(req) == -EINPROGRESS)
+		msleep(10);
+
+	memcpy(out, rctx, sizeof(*rctx) + SHA512_BLOCK_SIZE);
+
+	return 0;
+}
+
+static int omap_sham_import(struct ahash_request *req, const void *in)
+{
+	struct omap_sham_reqctx *rctx = ahash_request_ctx(req);
+
+	memcpy(rctx, in, sizeof(*rctx) + SHA512_BLOCK_SIZE);
+
+	return 0;
+}
+
 static struct ahash_alg algs_sha1_md5[] = {
 {
 	.init		= omap_sham_init,

@@ -2037,8 +2058,14 @@ static int omap_sham_probe(struct platform_device *pdev)
 
 	for (i = 0; i < dd->pdata->algs_info_size; i++) {
 		for (j = 0; j < dd->pdata->algs_info[i].size; j++) {
-			err = crypto_register_ahash(
-					&dd->pdata->algs_info[i].algs_list[j]);
+			struct ahash_alg *alg;
+
+			alg = &dd->pdata->algs_info[i].algs_list[j];
+			alg->export = omap_sham_export;
+			alg->import = omap_sham_import;
+			alg->halg.statesize = sizeof(struct omap_sham_reqctx) +
+				SHA512_BLOCK_SIZE;
+			err = crypto_register_ahash(alg);
 			if (err)
 				goto err_algs;
 

-- 
1.9.1

From: Lokesh Vutla <redacted>

As setting up the DMA operations is quite costly, add software fallback
support for requests smaller than 200 bytes. This change gives some 10%
extra performance in ipsec use case.

Signed-off-by: Lokesh Vutla <redacted>
[t-kristo at ti.com: udpated against latest upstream, to use skcipher mainly]
Signed-off-by: Tero Kristo <redacted>
---
 drivers/crypto/Kconfig    |  3 +++
 drivers/crypto/omap-aes.c | 53 +++++++++++++++++++++++++++++++++++++++++++----
 2 files changed, 52 insertions(+), 4 deletions(-)

diff --git a/drivers/crypto/Kconfig b/drivers/crypto/Kconfig
index 1af94e2..19ee6ee 100644
--- a/drivers/crypto/Kconfig
+++ b/drivers/crypto/Kconfig

@@ -318,6 +318,9 @@ config CRYPTO_DEV_OMAP_AES
 	select CRYPTO_AES
 	select CRYPTO_BLKCIPHER
 	select CRYPTO_ENGINE
+	select CRYPTO_CBC
+	select CRYPTO_ECB
+	select CRYPTO_CTR
 	help
 	  OMAP processors have AES module accelerator. Select this if you
 	  want to use the OMAP module for AES algorithms.

diff --git a/drivers/crypto/omap-aes.c b/drivers/crypto/omap-aes.c
index 3cf7b8f..6ba4f70 100644
--- a/drivers/crypto/omap-aes.c
+++ b/drivers/crypto/omap-aes.c

@@ -35,7 +35,7 @@
 #include <linux/interrupt.h>
 #include <crypto/scatterwalk.h>
 #include <crypto/aes.h>
-#include <crypto/algapi.h>
+#include <crypto/internal/skcipher.h>
 
 #define DST_MAXBURST			4
 #define DMA_MIN				(DST_MAXBURST * sizeof(u32))

@@ -105,6 +105,7 @@ struct omap_aes_ctx {
 	int		keylen;
 	u32		key[AES_KEYSIZE_256 / sizeof(u32)];
 	unsigned long	flags;
+	struct crypto_skcipher	*fallback;
 };
 
 struct omap_aes_reqctx {

@@ -701,11 +702,29 @@ static int omap_aes_crypt(struct ablkcipher_request *req, unsigned long mode)
 			crypto_ablkcipher_reqtfm(req));
 	struct omap_aes_reqctx *rctx = ablkcipher_request_ctx(req);
 	struct omap_aes_dev *dd;
+	int ret;
 
 	pr_debug("nbytes: %d, enc: %d, cbc: %d\n", req->nbytes,
 		  !!(mode & FLAGS_ENCRYPT),
 		  !!(mode & FLAGS_CBC));
 
+	if (req->nbytes < 200) {
+		SKCIPHER_REQUEST_ON_STACK(subreq, ctx->fallback);
+
+		skcipher_request_set_tfm(subreq, ctx->fallback);
+		skcipher_request_set_callback(subreq, req->base.flags, NULL,
+					      NULL);
+		skcipher_request_set_crypt(subreq, req->src, req->dst,
+					   req->nbytes, req->info);
+
+		if (mode & FLAGS_ENCRYPT)
+			ret = crypto_skcipher_encrypt(subreq);
+		else
+			ret = crypto_skcipher_decrypt(subreq);
+
+		skcipher_request_zero(subreq);
+		return ret;
+	}
 	dd = omap_aes_find_dev(ctx);
 	if (!dd)
 		return -ENODEV;

@@ -721,6 +740,7 @@ static int omap_aes_setkey(struct crypto_ablkcipher *tfm, const u8 *key,
 			   unsigned int keylen)
 {
 	struct omap_aes_ctx *ctx = crypto_ablkcipher_ctx(tfm);
+	int ret;
 
 	if (keylen != AES_KEYSIZE_128 && keylen != AES_KEYSIZE_192 &&
 		   keylen != AES_KEYSIZE_256)

@@ -731,6 +751,14 @@ static int omap_aes_setkey(struct crypto_ablkcipher *tfm, const u8 *key,
 	memcpy(ctx->key, key, keylen);
 	ctx->keylen = keylen;
 
+	crypto_skcipher_clear_flags(ctx->fallback, CRYPTO_TFM_REQ_MASK);
+	crypto_skcipher_set_flags(ctx->fallback, tfm->base.crt_flags &
+						 CRYPTO_TFM_REQ_MASK);
+
+	ret = crypto_skcipher_setkey(ctx->fallback, key, keylen);
+	if (!ret)
+		return 0;
+
 	return 0;
 }
 

@@ -766,6 +794,17 @@ static int omap_aes_ctr_decrypt(struct ablkcipher_request *req)
 
 static int omap_aes_cra_init(struct crypto_tfm *tfm)
 {
+	const char *name = crypto_tfm_alg_name(tfm);
+	const u32 flags = CRYPTO_ALG_ASYNC | CRYPTO_ALG_NEED_FALLBACK;
+	struct omap_aes_ctx *ctx = crypto_tfm_ctx(tfm);
+	struct crypto_skcipher *blk;
+
+	blk = crypto_alloc_skcipher(name, 0, flags);
+	if (IS_ERR(blk))
+		return PTR_ERR(blk);
+
+	ctx->fallback = blk;
+
 	tfm->crt_ablkcipher.reqsize = sizeof(struct omap_aes_reqctx);
 
 	return 0;

@@ -773,6 +812,12 @@ static int omap_aes_cra_init(struct crypto_tfm *tfm)
 
 static void omap_aes_cra_exit(struct crypto_tfm *tfm)
 {
+	struct omap_aes_ctx *ctx = crypto_tfm_ctx(tfm);
+
+	if (ctx->fallback)
+		crypto_free_skcipher(ctx->fallback);
+
+	ctx->fallback = NULL;
 }
 
 /* ********************** ALGS ************************************ */

@@ -784,7 +829,7 @@ static struct crypto_alg algs_ecb_cbc[] = {
 	.cra_priority		= 300,
 	.cra_flags		= CRYPTO_ALG_TYPE_ABLKCIPHER |
 				  CRYPTO_ALG_KERN_DRIVER_ONLY |
-				  CRYPTO_ALG_ASYNC,
+				  CRYPTO_ALG_ASYNC | CRYPTO_ALG_NEED_FALLBACK,
 	.cra_blocksize		= AES_BLOCK_SIZE,
 	.cra_ctxsize		= sizeof(struct omap_aes_ctx),
 	.cra_alignmask		= 0,

@@ -806,7 +851,7 @@ static struct crypto_alg algs_ecb_cbc[] = {
 	.cra_priority		= 300,
 	.cra_flags		= CRYPTO_ALG_TYPE_ABLKCIPHER |
 				  CRYPTO_ALG_KERN_DRIVER_ONLY |
-				  CRYPTO_ALG_ASYNC,
+				  CRYPTO_ALG_ASYNC | CRYPTO_ALG_NEED_FALLBACK,
 	.cra_blocksize		= AES_BLOCK_SIZE,
 	.cra_ctxsize		= sizeof(struct omap_aes_ctx),
 	.cra_alignmask		= 0,

@@ -832,7 +877,7 @@ static struct crypto_alg algs_ctr[] = {
 	.cra_priority		= 300,
 	.cra_flags		= CRYPTO_ALG_TYPE_ABLKCIPHER |
 				  CRYPTO_ALG_KERN_DRIVER_ONLY |
-				  CRYPTO_ALG_ASYNC,
+				  CRYPTO_ALG_ASYNC | CRYPTO_ALG_NEED_FALLBACK,
 	.cra_blocksize		= AES_BLOCK_SIZE,
 	.cra_ctxsize		= sizeof(struct omap_aes_ctx),
 	.cra_alignmask		= 0,

-- 
1.9.1

Calling runtime PM API at the cra_init/exit is bad for power management
purposes, as the lifetime for a CRA can be very long. Instead, use
pm_runtime autosuspend approach for handling the device clocks. Clocks
are enabled when they are actually required, and autosuspend disables
these if they have not been used for a sufficiently long time period.
By default, the timeout value is 1 second.

Signed-off-by: Tero Kristo <redacted>
---
 drivers/crypto/omap-aes.c | 43 ++++++++++++++++---------------------------
 1 file changed, 16 insertions(+), 27 deletions(-)

diff --git a/drivers/crypto/omap-aes.c b/drivers/crypto/omap-aes.c
index 4ab53a6..f443042 100644
--- a/drivers/crypto/omap-aes.c
+++ b/drivers/crypto/omap-aes.c

@@ -85,6 +85,8 @@
 #define AES_REG_IRQ_DATA_OUT           BIT(2)
 #define DEFAULT_TIMEOUT		(5*HZ)
 
+#define DEFAULT_AUTOSUSPEND_DELAY	1000
+
 #define FLAGS_MODE_MASK		0x000f
 #define FLAGS_ENCRYPT		BIT(0)
 #define FLAGS_CBC		BIT(1)

@@ -238,11 +240,19 @@ static void omap_aes_write_n(struct omap_aes_dev *dd, u32 offset,
 
 static int omap_aes_hw_init(struct omap_aes_dev *dd)
 {
+	int err;
+
 	if (!(dd->flags & FLAGS_INIT)) {
 		dd->flags |= FLAGS_INIT;
 		dd->err = 0;
 	}
 
+	err = pm_runtime_get_sync(dd->dev);
+	if (err < 0) {
+		dev_err(dd->dev, "failed to get sync: %d\n", err);
+		return err;
+	}
+
 	return 0;
 }
 

@@ -520,6 +530,9 @@ static void omap_aes_finish_req(struct omap_aes_dev *dd, int err)
 	pr_debug("err: %d\n", err);
 
 	crypto_finalize_request(dd->engine, req, err);
+
+	pm_runtime_mark_last_busy(dd->dev);
+	pm_runtime_put_autosuspend(dd->dev);
 }
 
 static int omap_aes_crypt_dma_stop(struct omap_aes_dev *dd)

@@ -761,23 +774,6 @@ static int omap_aes_ctr_decrypt(struct ablkcipher_request *req)
 
 static int omap_aes_cra_init(struct crypto_tfm *tfm)
 {
-	struct omap_aes_dev *dd = NULL;
-	int err;
-
-	/* Find AES device, currently picks the first device */
-	spin_lock_bh(&list_lock);
-	list_for_each_entry(dd, &dev_list, list) {
-		break;
-	}
-	spin_unlock_bh(&list_lock);
-
-	err = pm_runtime_get_sync(dd->dev);
-	if (err < 0) {
-		dev_err(dd->dev, "%s: failed to get_sync(%d)\n",
-			__func__, err);
-		return err;
-	}
-
 	tfm->crt_ablkcipher.reqsize = sizeof(struct omap_aes_reqctx);
 
 	return 0;

@@ -785,16 +781,6 @@ static int omap_aes_cra_init(struct crypto_tfm *tfm)
 
 static void omap_aes_cra_exit(struct crypto_tfm *tfm)
 {
-	struct omap_aes_dev *dd = NULL;
-
-	/* Find AES device, currently picks the first device */
-	spin_lock_bh(&list_lock);
-	list_for_each_entry(dd, &dev_list, list) {
-		break;
-	}
-	spin_unlock_bh(&list_lock);
-
-	pm_runtime_put_sync(dd->dev);
 }
 
 /* ********************** ALGS ************************************ */

@@ -1140,6 +1126,9 @@ static int omap_aes_probe(struct platform_device *pdev)
 	}
 	dd->phys_base = res.start;
 
+	pm_runtime_use_autosuspend(dev);
+	pm_runtime_set_autosuspend_delay(dev, DEFAULT_AUTOSUSPEND_DELAY);
+
 	pm_runtime_enable(dev);
 	err = pm_runtime_get_sync(dev);
 	if (err < 0) {

-- 
1.9.1

The crypto engine must be initialized before registering algorithms,
otherwise the test manager will crash as it attempts to execute
tests for the algos while they are being registered.

Fixes: 0529900a01cb ("crypto: omap-aes - Support crypto engine framework")
Signed-off-by: Tero Kristo <redacted>
---
 drivers/crypto/omap-aes.c | 28 +++++++++++++++-------------
 1 file changed, 15 insertions(+), 13 deletions(-)

diff --git a/drivers/crypto/omap-aes.c b/drivers/crypto/omap-aes.c
index 6ba4f70..8159523 100644
--- a/drivers/crypto/omap-aes.c
+++ b/drivers/crypto/omap-aes.c

@@ -1212,6 +1212,17 @@ static int omap_aes_probe(struct platform_device *pdev)
 	list_add_tail(&dd->list, &dev_list);
 	spin_unlock(&list_lock);
 
+	/* Initialize crypto engine */
+	dd->engine = crypto_engine_alloc_init(dev, 1);
+	if (!dd->engine)
+		goto err_engine;
+
+	dd->engine->prepare_request = omap_aes_prepare_req;
+	dd->engine->crypt_one_request = omap_aes_crypt_req;
+	err = crypto_engine_start(dd->engine);
+	if (err)
+		goto err_engine;
+
 	for (i = 0; i < dd->pdata->algs_info_size; i++) {
 		if (!dd->pdata->algs_info[i].registered) {
 			for (j = 0; j < dd->pdata->algs_info[i].size; j++) {

@@ -1229,26 +1240,17 @@ static int omap_aes_probe(struct platform_device *pdev)
 		}
 	}
 
-	/* Initialize crypto engine */
-	dd->engine = crypto_engine_alloc_init(dev, 1);
-	if (!dd->engine)
-		goto err_algs;
-
-	dd->engine->prepare_request = omap_aes_prepare_req;
-	dd->engine->crypt_one_request = omap_aes_crypt_req;
-	err = crypto_engine_start(dd->engine);
-	if (err)
-		goto err_engine;
-
 	return 0;
-err_engine:
-	crypto_engine_exit(dd->engine);
 err_algs:
 	for (i = dd->pdata->algs_info_size - 1; i >= 0; i--)
 		for (j = dd->pdata->algs_info[i].registered - 1; j >= 0; j--)
 			crypto_unregister_alg(
 					&dd->pdata->algs_info[i].algs_list[j]);
 
+err_engine:
+	if (dd->engine)
+		crypto_engine_exit(dd->engine);
+
 	omap_aes_dma_cleanup(dd);
 err_irq:
 	tasklet_kill(&dd->done_task);

-- 
1.9.1

The crypto engine must be initialized before registering algorithms,
otherwise the test manager will crash as it attempts to execute
tests for the algos while they are being registered.

Fixes: f1b77aaca85a ("crypto: omap-des - Integrate with the crypto engine framework")
Signed-off-by: Tero Kristo <redacted>
---
 drivers/crypto/omap-des.c | 28 +++++++++++++++-------------
 1 file changed, 15 insertions(+), 13 deletions(-)

diff --git a/drivers/crypto/omap-des.c b/drivers/crypto/omap-des.c
index d0b59f6..2b07e60 100644
--- a/drivers/crypto/omap-des.c
+++ b/drivers/crypto/omap-des.c

@@ -1085,6 +1085,17 @@ static int omap_des_probe(struct platform_device *pdev)
 	list_add_tail(&dd->list, &dev_list);
 	spin_unlock(&list_lock);
 
+	/* Initialize des crypto engine */
+	dd->engine = crypto_engine_alloc_init(dev, 1);
+	if (!dd->engine)
+		goto err_engine;
+
+	dd->engine->prepare_request = omap_des_prepare_req;
+	dd->engine->crypt_one_request = omap_des_crypt_req;
+	err = crypto_engine_start(dd->engine);
+	if (err)
+		goto err_engine;
+
 	for (i = 0; i < dd->pdata->algs_info_size; i++) {
 		for (j = 0; j < dd->pdata->algs_info[i].size; j++) {
 			algp = &dd->pdata->algs_info[i].algs_list[j];

@@ -1100,27 +1111,18 @@ static int omap_des_probe(struct platform_device *pdev)
 		}
 	}
 
-	/* Initialize des crypto engine */
-	dd->engine = crypto_engine_alloc_init(dev, 1);
-	if (!dd->engine)
-		goto err_algs;
-
-	dd->engine->prepare_request = omap_des_prepare_req;
-	dd->engine->crypt_one_request = omap_des_crypt_req;
-	err = crypto_engine_start(dd->engine);
-	if (err)
-		goto err_engine;
-
 	return 0;
 
-err_engine:
-	crypto_engine_exit(dd->engine);
 err_algs:
 	for (i = dd->pdata->algs_info_size - 1; i >= 0; i--)
 		for (j = dd->pdata->algs_info[i].registered - 1; j >= 0; j--)
 			crypto_unregister_alg(
 					&dd->pdata->algs_info[i].algs_list[j]);
 
+err_engine:
+	if (dd->engine)
+		crypto_engine_exit(dd->engine);
+
 	omap_des_dma_cleanup(dd);
 err_irq:
 	tasklet_kill(&dd->done_task);

-- 
1.9.1

If software fallback is used on older hardware accelerator setup (OMAP2/
OMAP3), the first block of data must be purged from the buffer. The
first block contains the pre-generated ipad value required by the HW,
but the software fallback algorithm generates its own, causing wrong
results.

Signed-off-by: Tero Kristo <redacted>
---
 drivers/crypto/omap-sham.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/drivers/crypto/omap-sham.c b/drivers/crypto/omap-sham.c
index b4f5131..f9c5fe5 100644
--- a/drivers/crypto/omap-sham.c
+++ b/drivers/crypto/omap-sham.c

@@ -1145,9 +1145,20 @@ static int omap_sham_final_shash(struct ahash_request *req)
 {
 	struct omap_sham_ctx *tctx = crypto_tfm_ctx(req->base.tfm);
 	struct omap_sham_reqctx *ctx = ahash_request_ctx(req);
+	int offset = 0;
+
+	/*
+	 * If we are running HMAC on limited hardware support, skip
+	 * the ipad in the beginning of the buffer if we are going for
+	 * software fallback algorithm.
+	 */
+	if (test_bit(FLAGS_HMAC, &ctx->flags) &&
+	    !test_bit(FLAGS_AUTO_XOR, &ctx->dd->flags))
+		offset = get_block_size(ctx);
 
 	return omap_sham_shash_digest(tctx->fallback, req->base.flags,
-				      ctx->buffer, ctx->bufcnt, req->result);
+				      ctx->buffer + offset,
+				      ctx->bufcnt - offset, req->result);
 }
 
 static int omap_sham_final(struct ahash_request *req)

-- 
1.9.1

From: Lokesh Vutla <redacted>

For cases where total length of an input SGs is not same as
length of the input data for encryption, omap-des driver
crashes. This happens in the case when IPsec is trying to use
omap-des driver.

To avoid this, we copy all the pages from the input SG list
into a contiguous buffer and prepare a single element SG list
for this buffer with length as the total bytes to crypt, which is
similar thing that is done in case of unaligned lengths.

Signed-off-by: Lokesh Vutla <redacted>
Tested-by: Aparna Balasubramanian <redacted>
Signed-off-by: Tero Kristo <redacted>
---
 drivers/crypto/omap-des.c | 27 +++++++++++++++++----------
 1 file changed, 17 insertions(+), 10 deletions(-)

diff --git a/drivers/crypto/omap-des.c b/drivers/crypto/omap-des.c
index 5691434..d0b59f6 100644
--- a/drivers/crypto/omap-des.c
+++ b/drivers/crypto/omap-des.c

@@ -521,29 +521,36 @@ static int omap_des_crypt_dma_stop(struct omap_des_dev *dd)
 	return 0;
 }
 
-static int omap_des_copy_needed(struct scatterlist *sg)
+static int omap_des_copy_needed(struct scatterlist *sg, int total)
 {
+	int len = 0;
+
+	if (!IS_ALIGNED(total, DES_BLOCK_SIZE))
+		return -1;
+
 	while (sg) {
 		if (!IS_ALIGNED(sg->offset, 4))
 			return -1;
 		if (!IS_ALIGNED(sg->length, DES_BLOCK_SIZE))
 			return -1;
+
+		len += sg->length;
 		sg = sg_next(sg);
 	}
+
+	if (len != total)
+		return -1;
+
 	return 0;
 }
 
 static int omap_des_copy_sgs(struct omap_des_dev *dd)
 {
 	void *buf_in, *buf_out;
-	int pages;
-
-	pages = dd->total >> PAGE_SHIFT;
-
-	if (dd->total & (PAGE_SIZE-1))
-		pages++;
+	int pages, total;
 
-	BUG_ON(!pages);
+	total = ALIGN(dd->total, DES_BLOCK_SIZE);
+	pages = get_order(total);
 
 	buf_in = (void *)__get_free_pages(GFP_ATOMIC, pages);
 	buf_out = (void *)__get_free_pages(GFP_ATOMIC, pages);

@@ -605,8 +612,8 @@ static int omap_des_prepare_req(struct crypto_engine *engine,
 	if (dd->out_sg_len < 0)
 		return dd->out_sg_len;
 
-	if (omap_des_copy_needed(dd->in_sg) ||
-	    omap_des_copy_needed(dd->out_sg)) {
+	if (omap_des_copy_needed(dd->in_sg, dd->total) ||
+	    omap_des_copy_needed(dd->out_sg, dd->total)) {
 		if (omap_des_copy_sgs(dd))
 			pr_err("Failed to copy SGs for unaligned cases\n");
 		dd->sgs_copied = 1;

-- 
1.9.1

If we have processed any data with the hardware accelerator (digcnt > 0),
we must complete the entire hash by using it. This is because the current
hash value can't be imported to the software fallback algorithm. Otherwise
we end up with wrong hash results.

Signed-off-by: Tero Kristo <redacted>
---
 drivers/crypto/omap-sham.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/crypto/omap-sham.c b/drivers/crypto/omap-sham.c
index aa71e61..b4f5131 100644
--- a/drivers/crypto/omap-sham.c
+++ b/drivers/crypto/omap-sham.c

@@ -1165,7 +1165,7 @@ static int omap_sham_final(struct ahash_request *req)
 	 * If buffersize is less than 240, we use fallback SW encoding,
 	 * as using DMA + HW in this case doesn't provide any benefit.
 	 */
-	if ((ctx->digcnt + ctx->bufcnt) < 240)
+	if (!ctx->digcnt && ctx->bufcnt < 240)
 		return omap_sham_final_shash(req);
 	else if (ctx->bufcnt)
 		return omap_sham_enqueue(req, OP_FINAL);

-- 
1.9.1

From: Lokesh Vutla <redacted>

Some SoCs like omap4/omap5/dra7 contain multiple AES crypto accelerator
cores. Adapt the driver to support this. The driver picks the last used
device from a list of AES devices.

Signed-off-by: Lokesh Vutla <redacted>
[t-kristo at ti.com: forward ported to 4.7 kernel]
Signed-off-by: Tero Kristo <redacted>
---
 drivers/crypto/omap-aes.c | 20 ++++++--------------
 1 file changed, 6 insertions(+), 14 deletions(-)

diff --git a/drivers/crypto/omap-aes.c b/drivers/crypto/omap-aes.c
index f443042..3cf7b8f 100644
--- a/drivers/crypto/omap-aes.c
+++ b/drivers/crypto/omap-aes.c

@@ -329,20 +329,12 @@ static void omap_aes_dma_stop(struct omap_aes_dev *dd)
 
 static struct omap_aes_dev *omap_aes_find_dev(struct omap_aes_ctx *ctx)
 {
-	struct omap_aes_dev *dd = NULL, *tmp;
+	struct omap_aes_dev *dd;
 
 	spin_lock_bh(&list_lock);
-	if (!ctx->dd) {
-		list_for_each_entry(tmp, &dev_list, list) {
-			/* FIXME: take fist available aes core */
-			dd = tmp;
-			break;
-		}
-		ctx->dd = dd;
-	} else {
-		/* already found before */
-		dd = ctx->dd;
-	}
+	dd = list_first_entry(&dev_list, struct omap_aes_dev, list);
+	list_move_tail(&dd->list, &dev_list);
+	ctx->dd = dd;
 	spin_unlock_bh(&list_lock);
 
 	return dd;

@@ -615,7 +607,7 @@ static int omap_aes_prepare_req(struct crypto_engine *engine,
 {
 	struct omap_aes_ctx *ctx = crypto_ablkcipher_ctx(
 			crypto_ablkcipher_reqtfm(req));
-	struct omap_aes_dev *dd = omap_aes_find_dev(ctx);
+	struct omap_aes_dev *dd = ctx->dd;
 	struct omap_aes_reqctx *rctx;
 
 	if (!dd)

@@ -661,7 +653,7 @@ static int omap_aes_crypt_req(struct crypto_engine *engine,
 {
 	struct omap_aes_ctx *ctx = crypto_ablkcipher_ctx(
 			crypto_ablkcipher_reqtfm(req));
-	struct omap_aes_dev *dd = omap_aes_find_dev(ctx);
+	struct omap_aes_dev *dd = ctx->dd;
 
 	if (!dd)
 		return -ENODEV;

-- 
1.9.1

On Thu, Aug 04, 2016 at 01:28:38PM +0300, Tero Kristo wrote:

Do we really need this? You must not call export until the previous
operation has completed.

Cheers,
-- 
Email: Herbert Xu [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

On 09/08/16 13:06, Herbert Xu wrote:

On Thu, Aug 04, 2016 at 01:28:38PM +0300, Tero Kristo wrote:

quoted

Context export/import are now required for ahash algorithms due to
required support in algif_hash. Implement these for OMAP SHA driver,
saving and restoring the internal state of the driver.

Signed-off-by: Tero Kristo <redacted>
---
 drivers/crypto/omap-sham.c | 31 +++++++++++++++++++++++++++++--
 1 file changed, 29 insertions(+), 2 deletions(-)

diff --git a/drivers/crypto/omap-sham.c b/drivers/crypto/omap-sham.c
index 6e53944..aa71e61 100644
--- a/drivers/crypto/omap-sham.c
+++ b/drivers/crypto/omap-sham.c

@@ -1379,6 +1379,27 @@ exit_unlock:
 	return ret;
 }

+static int omap_sham_export(struct ahash_request *req, void *out)
+{
+	struct omap_sham_reqctx *rctx = ahash_request_ctx(req);
+
+	while (omap_sham_flush(req) == -EINPROGRESS)
+		msleep(10);

Do we really need this? You must not call export until the previous
operation has completed.

Cheers,

Sorry about a late reply, I was out on vacation.

For OMAP SHAM, this is actually needed, because the driver still has a 
very large internal buffer for performance reasons, and the whole buffer 
can't be exported. The flush functionality pushes out sufficient amount 
of data to the hardware, so that the rest of the buffer can be exported 
to the available space.

This is pretty much related to the discussion we had previously here:

https://patchwork.kernel.org/patch/9192881/

Basically I decided to keep the driver buffer the same size as 
previously, but flush out any extra data.

-Tero

On Mon, Aug 29, 2016 at 05:11:35PM +0300, Tero Kristo wrote:

quoted

quoted

+static int omap_sham_export(struct ahash_request *req, void *out)
+{
+	struct omap_sham_reqctx *rctx = ahash_request_ctx(req);
+
+	while (omap_sham_flush(req) == -EINPROGRESS)
+		msleep(10);

Do we really need this? You must not call export until the previous
operation has completed.

Cheers,

Sorry about a late reply, I was out on vacation.

For OMAP SHAM, this is actually needed, because the driver still has
a very large internal buffer for performance reasons, and the whole
buffer can't be exported. The flush functionality pushes out
sufficient amount of data to the hardware, so that the rest of the
buffer can be exported to the available space.

It doesn't matter whether you have a buffer or not.  The point
is that the completion function should not be called until the
operation is actually complete.  This is the whole point of the
async interface.

As the user must not call export until the completion function
has been called, there should be no need to wait in the export
function.

Cheers,
-- 
Email: Herbert Xu [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

On 01/09/16 06:33, Herbert Xu wrote:

On Mon, Aug 29, 2016 at 05:11:35PM +0300, Tero Kristo wrote:

quoted

quoted

quoted

+static int omap_sham_export(struct ahash_request *req, void *out)
+{
+	struct omap_sham_reqctx *rctx = ahash_request_ctx(req);
+
+	while (omap_sham_flush(req) == -EINPROGRESS)
+		msleep(10);

Do we really need this? You must not call export until the previous
operation has completed.

Cheers,

Sorry about a late reply, I was out on vacation.

For OMAP SHAM, this is actually needed, because the driver still has
a very large internal buffer for performance reasons, and the whole
buffer can't be exported. The flush functionality pushes out
sufficient amount of data to the hardware, so that the rest of the
buffer can be exported to the available space.

It doesn't matter whether you have a buffer or not.  The point
is that the completion function should not be called until the
operation is actually complete.  This is the whole point of the
async interface.

As the user must not call export until the completion function
has been called, there should be no need to wait in the export
function.

Well, but the driver doesn't flush its buffers automatically, it caches 
data until it has sufficient amount available. So, assuming you want to 
do this:

sham_init
   sham_update 256 bytes
   sham_update 256 bytes
   wait until two above updates are complete
   sham_export

... the execution hangs at the wait phase as the driver is still waiting 
for more data to cache, and will never complete the two update requests. 
Currently, the driver is written in such way that it waits until it has 
enough data cached before starting to push it out to hardware, or waits 
until sham_final to be called. Pushing out small pieces of data causes 
severe performance degradation on the driver, as setting up the DMA 
operation itself is rather costly.

Either way, flush for the buffers is needed, I wonder if automatic flush 
should be added also based on some timer.

-Tero

On Thu, Sep 01, 2016 at 09:12:59AM +0300, Tero Kristo wrote:

Well, but the driver doesn't flush its buffers automatically, it
caches data until it has sufficient amount available. So, assuming
you want to do this:

sham_init
  sham_update 256 bytes
  sham_update 256 bytes
  wait until two above updates are complete
  sham_export

... the execution hangs at the wait phase as the driver is still

Well that's a bug in the driver.  While it's not illegal to wait
for more data, it's usually unnecessary.  Because we instead try
to get our users to generate as big a request as possible, e.g.,
one packet for IPsec.

If you really have to do the hold thing, then you must install a
timer like sha1-mb does on x86 to do the flush.

In any case, the completion function must not be called until
you're actually complete.

Cheers,
-- 
Email: Herbert Xu [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

On 01/09/16 09:16, Herbert Xu wrote:

On Thu, Sep 01, 2016 at 09:12:59AM +0300, Tero Kristo wrote:

quoted

Well, but the driver doesn't flush its buffers automatically, it
caches data until it has sufficient amount available. So, assuming
you want to do this:

sham_init
  sham_update 256 bytes
  sham_update 256 bytes
  wait until two above updates are complete
  sham_export

... the execution hangs at the wait phase as the driver is still

Well that's a bug in the driver.  While it's not illegal to wait
for more data, it's usually unnecessary.  Because we instead try
to get our users to generate as big a request as possible, e.g.,
one packet for IPsec.

If you really have to do the hold thing, then you must install a
timer like sha1-mb does on x86 to do the flush.

In any case, the completion function must not be called until
you're actually complete.

Hmm, looking at the driver, sham_update returns 0 immediately if it just 
caches data. In a sense, the update is not completed at this point. Are 
you saying this is illegal and can't be done?

 From my understanding, valid results are expected from the driver only 
after ->final is called.

-Tero

On Thu, Sep 01, 2016 at 09:56:06AM +0300, Tero Kristo wrote:

Hmm, looking at the driver, sham_update returns 0 immediately if it
just caches data. In a sense, the update is not completed at this
point. Are you saying this is illegal and can't be done?

Once you call the completion function (and returning zero from
the update itself is equivalent to calling the completion function)
the hardware must not touch the request anymore.

From my understanding, valid results are expected from the driver
only after ->final is called.

That's because you never implemented export/import :)

Cheers,
-- 
Email: Herbert Xu [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

On 01/09/16 10:19, Herbert Xu wrote:

On Thu, Sep 01, 2016 at 09:56:06AM +0300, Tero Kristo wrote:

quoted

Hmm, looking at the driver, sham_update returns 0 immediately if it
just caches data. In a sense, the update is not completed at this
point. Are you saying this is illegal and can't be done?

Once you call the completion function (and returning zero from
the update itself is equivalent to calling the completion function)
the hardware must not touch the request anymore.

Yeah, it is not touching it anymore. All the data has been copied to the 
local buffer at this point, and the driver isn't retaining any kind of a 
handle to the request itself anymore. It is just that it has not been 
processed by the hardware yet.

quoted

From my understanding, valid results are expected from the driver
only after ->final is called.

That's because you never implemented export/import :)

Yeah, the flush should do the trick now. Kind of a chicken-egg problem 
here. :P How do you see the situation with the above explanation?


-Tero

On Thu, Sep 01, 2016 at 10:28:47AM +0300, Tero Kristo wrote:

Yeah, the flush should do the trick now. Kind of a chicken-egg
problem here. :P How do you see the situation with the above
explanation?

The export function is not allowed to sleep so you must not wait
for the hardware to complete in it.

If you need to wait, then you must use the completion mechanism.

Cheers,
-- 
Email: Herbert Xu [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

On 01/09/16 10:31, Herbert Xu wrote:

On Thu, Sep 01, 2016 at 10:28:47AM +0300, Tero Kristo wrote:

quoted

Yeah, the flush should do the trick now. Kind of a chicken-egg
problem here. :P How do you see the situation with the above
explanation?

The export function is not allowed to sleep so you must not wait
for the hardware to complete in it.

If you need to wait, then you must use the completion mechanism.

I don't think export is allowed to return -EINPROGRESS either? At least 
currently the kernel pieces using this functionality won't work if I do 
that.

If thats the case, I need to think of something else to handle this...

-Tero

On 01/09/16 10:46, Tero Kristo wrote:

On 01/09/16 10:31, Herbert Xu wrote:

quoted

On Thu, Sep 01, 2016 at 10:28:47AM +0300, Tero Kristo wrote:

quoted

Yeah, the flush should do the trick now. Kind of a chicken-egg
problem here. :P How do you see the situation with the above
explanation?

The export function is not allowed to sleep so you must not wait
for the hardware to complete in it.

If you need to wait, then you must use the completion mechanism.

I don't think export is allowed to return -EINPROGRESS either? At least
currently the kernel pieces using this functionality won't work if I do
that.

If thats the case, I need to think of something else to handle this...

-Tero

Hi Herbert,

Additional request, would it be possible for you to check the rest of 
the series and just ignore patches #2 and #3 for now, the rest don't 
have any dependencies against these and can be applied cleanly without.

I would like to see these move forward while I figure out how to handle 
the buffer / export+import...

-Tero

On Mon, Sep 05, 2016 at 03:06:05PM +0300, Tero Kristo wrote:

Additional request, would it be possible for you to check the rest
of the series and just ignore patches #2 and #3 for now, the rest
don't have any dependencies against these and can be applied cleanly
without.

I would like to see these move forward while I figure out how to
handle the buffer / export+import...

Sure I'll review them again.

Cheers,
-- 
Email: Herbert Xu [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

On Thu, Aug 04, 2016 at 01:28:41PM +0300, Tero Kristo wrote:

From: Lokesh Vutla <redacted>

For cases where total length of an input SGs is not same as
length of the input data for encryption, omap-des driver
crashes. This happens in the case when IPsec is trying to use
omap-des driver.

To avoid this, we copy all the pages from the input SG list
into a contiguous buffer and prepare a single element SG list
for this buffer with length as the total bytes to crypt, which is
similar thing that is done in case of unaligned lengths.

Ugh, that means copying every single packet, right?

So if it's just the SG list that's the problem, why don't you
copy that instead? That is, allocate a new SG list and set it
up so that there is no excess data.

Cheers,
-- 
Email: Herbert Xu [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

On Thu, Aug 04, 2016 at 01:28:35PM +0300, Tero Kristo wrote:

Hi,

This revision took quite a bit time to craft due to the rework needed
for sham buffer handling and export/import. I ended up implementing
a flush functionality for draining out the sham buffer when doing
export/import; just shrinking the buffer to sufficiently small size
impacted the performance with small data chunks too much so I dropped
this approach.

The series also fixes a couple of existing issues with omap2/omap3
hardware acceleration, I ran a full boot test / crypto manager
test suite on all boards accessible to me now.

Based on top of latest mainline, which is somewhere before 4.8-rc1
as of writing this, I am unable to rebase the series during the next
three weeks so wanted to get this out now. Targeted for 4.9 merge
window, some fixes could be picked up earlier though if needed.

I have applied patches 1,4-5,7-11.  Some of them didn't apply
cleanly so please check the result in my tree.

Thanks,
-- 
Email: Herbert Xu [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

On 13/09/16 15:38, Herbert Xu wrote:

On Thu, Aug 04, 2016 at 01:28:35PM +0300, Tero Kristo wrote:

quoted

Hi,

This revision took quite a bit time to craft due to the rework needed
for sham buffer handling and export/import. I ended up implementing
a flush functionality for draining out the sham buffer when doing
export/import; just shrinking the buffer to sufficiently small size
impacted the performance with small data chunks too much so I dropped
this approach.

The series also fixes a couple of existing issues with omap2/omap3
hardware acceleration, I ran a full boot test / crypto manager
test suite on all boards accessible to me now.

Based on top of latest mainline, which is somewhere before 4.8-rc1
as of writing this, I am unable to rebase the series during the next
three weeks so wanted to get this out now. Targeted for 4.9 merge
window, some fixes could be picked up earlier though if needed.

I have applied patches 1,4-5,7-11.  Some of them didn't apply
cleanly so please check the result in my tree.

Thanks,

Thanks Herbert,

I just gave a trial for your branch, and seems to be working for me. 
Also checked the patches you applied and they seem fine also.

I have also a new version of the sha buffer handling and export/import 
available now, but need to cleanup it quite a bit, and figure out how to 
split the patch properly.

  drivers/crypto/omap-sham.c | 532 
++++++++++++++++++++++++++-------------------

... It now uses sg for xmitting data where possible, and avoids the need 
for a large internal buffer. The buffer size is also now properly 
configurable, which can be used to overcome the performance issues if 
needed (this however, requires the max statesize hack within the 
crypto/ahash.c file, but thats fine.) I'll hopefully post this out maybe 
tomorrow, but its going to be targeted for 4.10 I believe due to the 
(ahem, rather) intrusive changes.

-Tero

On 13/09/16 12:35, Herbert Xu wrote:

On Thu, Aug 04, 2016 at 01:28:41PM +0300, Tero Kristo wrote:

quoted

From: Lokesh Vutla <redacted>

For cases where total length of an input SGs is not same as
length of the input data for encryption, omap-des driver
crashes. This happens in the case when IPsec is trying to use
omap-des driver.

To avoid this, we copy all the pages from the input SG list
into a contiguous buffer and prepare a single element SG list
for this buffer with length as the total bytes to crypt, which is
similar thing that is done in case of unaligned lengths.

Ugh, that means copying every single packet, right?

So if it's just the SG list that's the problem, why don't you
copy that instead? That is, allocate a new SG list and set it
up so that there is no excess data.

Cheers,

I'll take a look at this. I have this kind of solution in place for the 
re-worked SHA driver, so can probably re-use it here.

-Tero