-----Original Message-----
From: Ahmed Mansour [mailto:ahmed.mansour@nxp.com]
Sent: Thursday, February 15, 2018 7:51 PM
To: Trahe, Fiona <redacted>; Verma, Shally <redacted>; dev@dpdk.org
Cc: Athreya, Narayana Prasad <redacted>; Gupta, Ashish
[off-list ref]; Sahu, Sunila [off-list ref]; De Lara Guarch, Pablo
[off-list ref]; Challa, Mahipal [off-list ref]; Jain, Deepak K
[off-list ref]; Hemant Agrawal [off-list ref]; Roy Pledge
[off-list ref]; Youri Querry [off-list ref]
Subject: Re: [RFC v2] doc compression API for DPDK

/// snip ///

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

[Fiona] I propose if BFINAL bit is detected before end of input
the decompression should stop. In this case consumed will be < src.length.
produced will be < dst buffer size. Do we need an extra STATUS response?
STATUS_BFINAL_DETECTED  ?

[Shally] @fiona, I assume you mean here decompressor stop after processing Final block right?

[Fiona] Yes.

 And if yes,

quoted

and if it can process that final block successfully/unsuccessfully, then status could simply be
SUCCESS/FAILED.
I don't see need of specific return code for this use case. Just to share, in past, we have

practically

quoted

quoted

run into

quoted

quoted

quoted

quoted

quoted

such cases with boost lib, and decompressor has simply worked this way.

[Fiona] I'm ok with this.

quoted

quoted

Only thing I don't like this is it can impact on performance, as normally
we can just look for STATUS == SUCCESS. Anything else should be an exception.
Now the application would have to check for SUCCESS || BFINAL_DETECTED every time.
Do you have a suggestion on how we should handle this?

[Ahmed] This makes sense. So in all cases the PMD should assume that it
should stop as soon as a BFINAL is observed.

A question. What happens ins stateful vs stateless modes when
decompressing an op that encompasses multiple BFINALs. I assume the
caller in that case will use the consumed=x bytes to find out how far in
to the input is the end of the first stream and start from the next
byte. Is this correct?

[Shally]  As per my understanding, each op can be tied up to only one stream as we have only one

stream pointer per op and one

quoted

stream can have only one BFINAL (as stream is one complete compressed data) but looks like you're

suggesting a case where one op

quoted

can carry multiple independent streams? and thus multiple BFINAL?! , such as, below here is op

pointing to more than one streams

quoted

quoted

            --------------------------------------------
op --> |stream1|stream2| |stream3|
           --------------------------------------------

Could you confirm if I understand your question correct?

[Ahmed] Correct. We found that in some storage applications the user
does not know where exactly the BFINAL is. They rely on zlib software
today. zlib.net software halts at the first BFINAL. Users put multiple
streams in one op and rely on zlib to  stop and inform them of the end
location of the first stream.

[Shally] Then this is practically case possible on decompressor and decompressor doesn't regard flush
flag. So in that case, I expect PMD to internally reset themselves (say in case of zlib going through

cycle

quoted

quoted

of deflateEnd and deflateInit or deflateReset) and return with status = SUCCESS with updated

produced

quoted

quoted

and consumed. Now in such case, if previous stream also has some footer followed by start of next
stream, then I am not sure how PMD / lib can support that case. Have you had practically run of such
use-case on zlib? If yes, how then such application handle it in your experience?
I can imagine for such input zlib would return with Z_FLUSH_END after 1st BFINAL is processed to the
user. Then application doing deflateReset() or Init-End() cycle before starting with next. But if it starts
with input that doesn't have valid zlib header, then likely it will throw an error.

[Fiona] The consumed and produced tell the Application hw much data was processed up to
the end of the first deflate block encountered with a bfinal set.
If there is data, e.g. footer after the block with bfinal, then I think it must be the responsibility of
the application to know this, the PMD can't have any responsibility for this.
The next op sent to the PMD must start with a valid deflate block.

[Ahmed] Agreed. This is exactly what I expected. In our case we support
gzip and zlib header/footer processing, but that does not fundamentally
change the setup. The user may have other meta data after the footer
which the PMD is not responsible for. The PMD should stop processing
depending on the mode. In raw DEFLATE, it should stop immediately. In
other modes it should stop after the footer. We also have a mode in our
PMD to simply continue decompression. In that case there cannot be
header/footer between streams in raw DEFLATE. That mode can be enabled
perhaps at the session level in the future with a session parameter at
setup time. We call it "member continue". In this mode the PMD plows
through as much of the op as possible. If it hits incorrectly setup data
then it returns what it did decompress successfully and the error code
in decompressing the data afterwards.

[Fiona] Yes, these would be interesting capabilities which could be 
added to the API in future releases.

quoted

quoted

quoted

quoted

Thanks
Shally