Previous hunk in different file during add --patch

I keep expecting that pressing K during patch add that it will cross
file boundaries and go to previous hunks in files before the current
one. However, it reports "no hunks" when reaching the top hunk in the
current file. Is there a way to go to the previous file?