I don't understand why the low_latency flag even exists when setting
it apparently breaks most all of the serial drivers.

Mostly history. I'd not realised random users could fiddle with it - they
shouldn't be able to do that so that wants fixing.

Historically it made sense as with a 100Hz clock some algorithms with
drivers that delayed stuff a lot got really slow (eg non windowed
versions of firmware download protocols, kermit, xmodem etc)

Nowdays the kernel is a bit more sophisticated and we should possibly
re-arrange it so that it selects different workqueues or similar -
low_latency meaning 'use a private hard rt work queue' perhaps.

Or indeed possibly making ttys use threaded IRQs