Re: [PATCH 5/6] xfrm_user: ensure user supplied esn replay window is valid

On Thu, Sep 20, 2012 at 8:12 AM, Mathias Krause [off-list ref] wrote:

What still might happen is the overflow in xfrm_replay_state_esn_len()
resulting in a to small bitmap allocation for the requested replay
size. But that gets catched in xfrm_init_replay(). Little late, but
hey.

Sorry, I mixed that up. The replay_window check in xfrm_init_replay()
has only little to do with the bmp_len overflow. But changing the
return type of xfrm_replay_state_esn_len() to size_t and by doing so,
making the all the size compares operating on positive values, we'll
at least allocate enough memory to not run into memory corruptions.
Though, the replay window will be much smaller, than requested -- due
to the overflow. But userland should expect this. A check for some
upper limit in verify_replay() could catch this early.

Mathias