Hi,
setserial has low_latency option which should minimize receive latency
(scheduler delay). AFAICT it is used if someone talk to external device
via RS-485/RS-232 and need to have quick requests and responses . On
kernel this feature was implemented by direct tty processing from
interrupt context:
void tty_flip_buffer_push(struct tty_port *port)
{
struct tty_bufhead *buf = &port->buf;
buf->tail->commit = buf->tail->used;
if (port->low_latency)
flush_to_ldisc(&buf->work);
else
schedule_work(&buf->work);
}
But after 3.12 tty locking changes, calling flush_to_ldisc() from
interrupt context is a bug (we got scheduling while atomic bug report
here: https://bugzilla.redhat.com/show_bug.cgi?id=1065087 )
I'm not sure how this should be solved. After Peter get rid all of those
race condition in tty layer, we probably don't want go back to use
spin_lock's there. Maybe we can create WQ_HIGHPRI workqueue and schedule
flush_to_ldisc() work there. Or perhaps users that need to low latency,
should switch to thread irq and prioritize serial irq to meat
retirements. Anyway setserial low_latency is now broken and all who use
this feature in the past can not do this any longer on 3.12+ kernels.
Thoughts ?
Stanislaw
From: One Thousand Gnomes <hidden> Date: 2014-02-18 09:57:54
spin_lock's there. Maybe we can create WQ_HIGHPRI workqueue and schedule
flush_to_ldisc() work there. Or perhaps users that need to low latency,
should switch to thread irq and prioritize serial irq to meat
retirements. Anyway setserial low_latency is now broken and all who use
this feature in the past can not do this any longer on 3.12+ kernels.
Historically speaking it was never allowed to use low_latency from a port
that did tty_flip_buffer_push from an IRQ as opposed to scheduling work.
The code also rather pre-dates threaded IRQ but that may well be a better
approach.
IMHO the right fix is to fastpath most of the tty layer (non N_TTY ldisc,
N_TTY without ICANON or ECHO*). Most of the remaining tty locking would
then go away almost entirely for these cases and we'd massively improve
things like our dismal 3G modem performance.
Likewise the termios lock can go by using RCU and passing the termios
struct into the driver as a copy of the RCU managed object (so we can
deal with sleeping drivers). Termios structs are tiny so the copying
overhead is basically nil.
It just needs someone sufficiently crazy and with a fair bit of time to
actually do the heavy lifting. I've been poking at bits of it but the
changes when switching ldisc are not entirely trivial and the N_TTY
fastpaths are quite a lot of work. Thankfully the non N_TTY ones are
simple.
Alan
From: Peter Hurley <hidden> Date: 2014-02-18 22:12:19
Hi Stanislaw,
On 02/18/2014 04:38 AM, Stanislaw Gruszka wrote:
Hi,
setserial has low_latency option which should minimize receive latency
(scheduler delay). AFAICT it is used if someone talk to external device
via RS-485/RS-232 and need to have quick requests and responses . On
kernel this feature was implemented by direct tty processing from
interrupt context:
void tty_flip_buffer_push(struct tty_port *port)
{
struct tty_bufhead *buf = &port->buf;
buf->tail->commit = buf->tail->used;
if (port->low_latency)
flush_to_ldisc(&buf->work);
else
schedule_work(&buf->work);
}
But after 3.12 tty locking changes, calling flush_to_ldisc() from
interrupt context is a bug (we got scheduling while atomic bug report
here: https://bugzilla.redhat.com/show_bug.cgi?id=1065087 )
I'm not sure how this should be solved. After Peter get rid all of those
race condition in tty layer, we probably don't want go back to use
spin_lock's there. Maybe we can create WQ_HIGHPRI workqueue and schedule
flush_to_ldisc() work there. Or perhaps users that need to low latency,
should switch to thread irq and prioritize serial irq to meat
retirements. Anyway setserial low_latency is now broken and all who use
this feature in the past can not do this any longer on 3.12+ kernels.
Thoughts ?
Can you give me an idea of your device's average and minimum required
latency (please be specific)? Is your target arch x86 [so I can evaluate the
the impact of bus-locked instructions relative to your expected]?
Also, how painful would it be if unsupported termios changes were rejected
if the port was in low_latency mode and/or if low_latency setting was
disallowed because of termios state?
It would be pointless to throttle low_latency, yes?
What would be an acceptable outcome of being unable to accept input?
Corrupted overrun? Dropped i/o? Queued for later? Please explain with
comparison to the outcome of missed minimum latency.
Regards,
Peter Hurley
Hello,
On Tue, Feb 18, 2014 at 05:12:13PM -0500, Peter Hurley wrote:
On 02/18/2014 04:38 AM, Stanislaw Gruszka wrote:
quoted
Hi,
setserial has low_latency option which should minimize receive latency
(scheduler delay). AFAICT it is used if someone talk to external device
via RS-485/RS-232 and need to have quick requests and responses . On
kernel this feature was implemented by direct tty processing from
interrupt context:
void tty_flip_buffer_push(struct tty_port *port)
{
struct tty_bufhead *buf = &port->buf;
buf->tail->commit = buf->tail->used;
if (port->low_latency)
flush_to_ldisc(&buf->work);
else
schedule_work(&buf->work);
}
But after 3.12 tty locking changes, calling flush_to_ldisc() from
interrupt context is a bug (we got scheduling while atomic bug report
here: https://bugzilla.redhat.com/show_bug.cgi?id=1065087 )
I'm not sure how this should be solved. After Peter get rid all of those
race condition in tty layer, we probably don't want go back to use
spin_lock's there. Maybe we can create WQ_HIGHPRI workqueue and schedule
flush_to_ldisc() work there. Or perhaps users that need to low latency,
should switch to thread irq and prioritize serial irq to meat
retirements. Anyway setserial low_latency is now broken and all who use
this feature in the past can not do this any longer on 3.12+ kernels.
Thoughts ?
Can you give me an idea of your device's average and minimum required
latency (please be specific)? Is your target arch x86 [so I can evaluate the
the impact of bus-locked instructions relative to your expected]?
Also, how painful would it be if unsupported termios changes were rejected
if the port was in low_latency mode and/or if low_latency setting was
disallowed because of termios state?
It would be pointless to throttle low_latency, yes?
What would be an acceptable outcome of being unable to accept input?
Corrupted overrun? Dropped i/o? Queued for later? Please explain with
comparison to the outcome of missed minimum latency.
I'm sorry, I can not answer your questions.
For what I googled it looks like users wanted to get rid of 10ms jitter
caused by scheduler. Unfortunately that is all I know. I'm not user of
this feature and I don't know what are exact expectations. I'm adding Hal,
who reported bug to RH bugzilla, perhaps he can tell more about what
expectations are.
For now, since issue is not easy fixable, perhaps we should just fix
scheduling while atomic bug and add warning like on below patch ?
Thanks
Stanislaw
@@ -504,9 +504,13 @@ void tty_flip_buffer_push(struct tty_port *port)buf->tail->commit=buf->tail->used;-if(port->low_latency)-flush_to_ldisc(&buf->work);-else+if(port->low_latency){+if(WARN_ONCE(in_irq()||irqs_disabled(),+"serial low_latency feature can not be used in interrupt context"))+schedule_work(&buf->work);+else+flush_to_ldisc(&buf->work);+}elseschedule_work(&buf->work);}EXPORT_SYMBOL(tty_flip_buffer_push);
From: Grant Edwards <hidden> Date: 2014-02-19 16:56:09
On 2014-02-19, Stanislaw Gruszka [off-list ref] wrote:
Hello,
On Tue, Feb 18, 2014 at 05:12:13PM -0500, Peter Hurley wrote:
quoted
On 02/18/2014 04:38 AM, Stanislaw Gruszka wrote:
quoted
setserial has low_latency option which should minimize receive latency
(scheduler delay). AFAICT it is used if someone talk to external device
via RS-485/RS-232 and need to have quick requests and responses.
Can you give me an idea of your device's average and minimum required
latency (please be specific)?
If by "device" you mean the UART being supported by the the driver in
question, then there is no answer.
Latency issues and requirements are a user-space application level
requirement that depend on the specific application's requirements and
those of the widget on the other end of the serial cable.
quoted
Also, how painful would it be if unsupported termios changes were rejected
if the port was in low_latency mode and/or if low_latency setting was
disallowed because of termios state?
It would be pointless to throttle low_latency, yes?
By "throttle" I assume you're talking about flow control?
_Usually_ applications that require low latency are exchanging short
messages (up to a few hundred bytes, but usually more like a few
dozen). In those cases flow control is not generally needed.
Does it matter?
quoted
What would be an acceptable outcome of being unable to accept input?
Corrupted overrun? Dropped i/o? Queued for later? Please explain with
comparison to the outcome of missed minimum latency.
I'm sorry, I can not answer your questions.
For what I googled it looks like users wanted to get rid of 10ms jitter
caused by scheduler.
Yes. That was the use for the low_latency option. Historically, all
of my drivers had supported the low_latency options for customers that
found scheduling delays to be a problem.
However, low_latency has been broken in certain contexts for a long
time. As a result, drivers have to avoid using it either completely or
sometimes only where rx data is handled in certain contexts.
For some drivers the end result of that is that you can choose either
a low-latency I/O mechanism or low-latency TTY layer rx handling, but
you can't use both at the same time because the low-latency I/O
mechanism handles rx data in a context where low-latency TTY layer
stuff can't be used.
Now that HZ is often 1000 and tickless is commonly used, I don't think
the scheduling delay is nearly as much an issue as it used to be. I
haven't gotten any complaints since it was largely rendered useless
several years ago.
Now all my drivers will silently override users if they set the
low_latency flag on a port in situations where it can't be used.
--
Grant Edwards grant.b.edwards Yow! OVER the underpass!
at UNDER the overpass!
gmail.com Around the FUTURE and
BEYOND REPAIR!!
From: Peter Hurley <hidden> Date: 2014-02-19 17:39:06
Hi Grant,
On 02/19/2014 11:55 AM, Grant Edwards wrote:
On 2014-02-19, Stanislaw Gruszka [off-list ref] wrote:
quoted
Hello,
On Tue, Feb 18, 2014 at 05:12:13PM -0500, Peter Hurley wrote:
quoted
On 02/18/2014 04:38 AM, Stanislaw Gruszka wrote:
quoted
setserial has low_latency option which should minimize receive latency
(scheduler delay). AFAICT it is used if someone talk to external device
via RS-485/RS-232 and need to have quick requests and responses.
Exactly.
But not exactly, because I need a quantified value for "quick",
preferably with average latency measurements for 3.11- and 3.12+
Can you give me an idea of your device's average and minimum required
latency (please be specific)?
If by "device" you mean the UART being supported by the the driver in
question, then there is no answer.
Latency issues and requirements are a user-space application level
requirement that depend on the specific application's requirements and
those of the widget on the other end of the serial cable.
I'm trying to determine if 3.12+ already satisfies the userspace requirement
(or if the requirement is infeasible).
The assumption is that 3.12+ w/o low_latency is worse than 3.11- w/ low_latency,
which may not be true.
Also, as you note, the latency requirement is in userspace, so bound
to the behavior of the scheduler anyway. Thus, immediately writing
to the read buffer from IRQ may have no different average latency than
handling by a worker (as measured by the elapsed time from interrupt to
userspace read).
quoted
quoted
Also, how painful would it be if unsupported termios changes were rejected
if the port was in low_latency mode and/or if low_latency setting was
disallowed because of termios state?
It would be pointless to throttle low_latency, yes?
By "throttle" I assume you're talking about flow control?
Driver throttle (but the discussion can include auto-flow control devices).
How can the requirement be for both must-handle-in-minimum-time data
(low_latency) and the-userspace-reader-isn't-reading-fast-enough-
so-its-ok-to-halt-transmission ?
Throttling/unthrottling the sender seems counter to "low latency".
_Usually_ applications that require low latency are exchanging short
messages (up to a few hundred bytes, but usually more like a few
dozen). In those cases flow control is not generally needed.
Does it matter?
Driver throttling requires excluding concurrent unthrottle and calling into
the driver (and said driver has relied on sleeping locks for many
kernel versions).
But first I'd like some hard data on whether or not a low latency
mode is even necessary (at least for user-space).
quoted
quoted
What would be an acceptable outcome of being unable to accept input?
Corrupted overrun? Dropped i/o? Queued for later? Please explain with
comparison to the outcome of missed minimum latency.
I'm sorry, I can not answer your questions.
For what I googled it looks like users wanted to get rid of 10ms jitter
caused by scheduler.
Yes. That was the use for the low_latency option. Historically, all
of my drivers had supported the low_latency options for customers that
found scheduling delays to be a problem.
However, low_latency has been broken in certain contexts for a long
time. As a result, drivers have to avoid using it either completely or
sometimes only where rx data is handled in certain contexts.
For some drivers the end result of that is that you can choose either
a low-latency I/O mechanism or low-latency TTY layer rx handling, but
you can't use both at the same time because the low-latency I/O
mechanism handles rx data in a context where low-latency TTY layer
stuff can't be used.
Now that HZ is often 1000 and tickless is commonly used, I don't think
the scheduling delay is nearly as much an issue as it used to be. I
haven't gotten any complaints since it was largely rendered useless
several years ago.
Now all my drivers will silently override users if they set the
low_latency flag on a port in situations where it can't be used.
Right. I'd rather not 'fix' something that doesn't really need
fixing (other than to suppress any WARNING caused by low_latency).
Regards,
Peter Hurley
From: Grant Edwards <hidden> Date: 2014-02-19 18:13:09
On 2014-02-19, Peter Hurley [off-list ref] wrote:
On 02/19/2014 11:55 AM, Grant Edwards wrote:
quoted
quoted
quoted
quoted
setserial has low_latency option which should minimize receive latency
(scheduler delay). AFAICT it is used if someone talk to external device
via RS-485/RS-232 and need to have quick requests and responses.
Exactly.
But not exactly, because I need a quantified value for "quick",
Only the end-user knows that, and they don't tell us what it is unless
it's not being met. :)
preferably with average latency measurements for 3.11- and 3.12+
It's been a long, long, time since I've done those measurements, and
whatever data I had is no longer relevent (even if I could find it) --
so I'll leave that up to somebody who's actually lobbying for getting
low_latency fixed so that it works from an interrupt context.
Just to be clearn, I'm not lobbying either for or against that. I'm
just trying to provide some perspective on the low_latency flag,
why it _was_ there, and who used it.
I'm trying to determine if 3.12+ already satisfies the userspace
requirement (or if the requirement is infeasible).
The assumption is that 3.12+ w/o low_latency is worse than 3.11- w/
low_latency, which may not be true.
I haven't heard any complaints for probably 10+ years. The last time
I did hear a complaint, I told them to set the low_latency flag and
that solved the problem.
Also, as you note, the latency requirement is in userspace, so bound
to the behavior of the scheduler anyway. Thus, immediately writing to
the read buffer from IRQ may have no different average latency than
handling by a worker (as measured by the elapsed time from interrupt
to userspace read).
Yes. And the low_latency flag used to (as in 10 years ago) have a big
effect on that. Without the low latency flag, the user read would
happen up to 10ms (assuming HZ=100) after the data was received by the
driver. Setting the low-latency flag eliminated that 10ms jitter. I
don't know if these days setting the low_latency flag (in contexts
where it does work) even has a noticeable effect.
How can the requirement be for both must-handle-in-minimum-time data
(low_latency) and the-userspace-reader-isn't-reading-fast-enough-
so-its-ok-to-halt-transmission ?
Throttling/unthrottling the sender seems counter to "low latency".
Agreed.
I can _imagine_ a case where an application has a strict limit how
long it will wait until it sees the first byte of the response, but
for some isolated cases (uploading large data logs) that response
might be very large -- large neough that the application may have to
pause while reading the response and rely on flow control to throttle
the upload. But, I can't point to any specific instance of that.
quoted
_Usually_ applications that require low latency are exchanging short
messages (up to a few hundred bytes, but usually more like a few
dozen). In those cases flow control is not generally needed.
Does it matter?
Driver throttling requires excluding concurrent unthrottle and
calling into the driver (and said driver has relied on sleeping locks
for many kernel versions).
Is that still an issue for drivers where the flow cotrol is handled by
the UART?
But first I'd like some hard data on whether or not a low latency
mode is even necessary (at least for user-space).
I don't have any hard data, but gut anser is that it is probably no
longer needed. The problem that existed for the past few years was
that there was a user-space way to set the low_latency flag (it didn't
require root and you could even do it from the command-line with
setserial) -- and doing so annoyed the kernel.
quoted
Now that HZ is often 1000 and tickless is commonly used, I don't think
the scheduling delay is nearly as much an issue as it used to be. I
haven't gotten any complaints since it was largely rendered useless
several years ago.
Now all my drivers will silently override users if they set the
low_latency flag on a port in situations where it can't be used.
Right. I'd rather not 'fix' something that doesn't really need fixing
(other than to suppress any WARNING caused by low_latency).
Yes. I think what currently needs to be done is to prevent any issues
caused by the user setting the low_latency. [IMO, warnings from the
kernel are an issue, even if the serial port continues to operate
properly.]
If somebody has specific latency requirements that aren't being met,
the current performance need to be measured, and it needs to be
decided if a solution is fesable and if the right solution is "fixing"
the low_latency flag so it's allowed in more contexts.
--
Grant Edwards grant.b.edwards Yow! I can't decide which
at WRONG TURN to make first!!
gmail.com I wonder if BOB GUCCIONE
has these problems!
From: Peter Hurley <hidden> Date: 2014-02-19 18:43:04
On 02/19/2014 01:12 PM, Grant Edwards wrote:
On 2014-02-19, Peter Hurley [off-list ref] wrote:
quoted
On 02/19/2014 11:55 AM, Grant Edwards wrote:
quoted
quoted
quoted
quoted
quoted
setserial has low_latency option which should minimize receive latency
(scheduler delay). AFAICT it is used if someone talk to external device
via RS-485/RS-232 and need to have quick requests and responses.
Exactly.
But not exactly, because I need a quantified value for "quick",
Only the end-user knows that, and they don't tell us what it is unless
it's not being met. :)
quoted
preferably with average latency measurements for 3.11- and 3.12+
It's been a long, long, time since I've done those measurements, and
whatever data I had is no longer relevent (even if I could find it) --
so I'll leave that up to somebody who's actually lobbying for getting
low_latency fixed so that it works from an interrupt context.
Just to be clearn, I'm not lobbying either for or against that. I'm
just trying to provide some perspective on the low_latency flag,
why it _was_ there, and who used it.
Yeah, I know; I didn't mean for you to supply those or champion the cause.
But since I'm being quoted on StackExchange, I thought I'd make my
expectations publicly known, so that if someone really feels they
need this feature, they'll speak up and back it up with numbers.
Actually right now, I'm looking into instrumenting the input path
with some event tracers so that I can actually measure that latency,
not in absolute terms, but at least relative to previous.
quoted
I'm trying to determine if 3.12+ already satisfies the userspace
requirement (or if the requirement is infeasible).
The assumption is that 3.12+ w/o low_latency is worse than 3.11- w/
low_latency, which may not be true.
I haven't heard any complaints for probably 10+ years. The last time
I did hear a complaint, I told them to set the low_latency flag and
that solved the problem.
quoted
Also, as you note, the latency requirement is in userspace, so bound
to the behavior of the scheduler anyway. Thus, immediately writing to
the read buffer from IRQ may have no different average latency than
handling by a worker (as measured by the elapsed time from interrupt
to userspace read).
Yes. And the low_latency flag used to (as in 10 years ago) have a big
effect on that. Without the low latency flag, the user read would
happen up to 10ms (assuming HZ=100) after the data was received by the
driver. Setting the low-latency flag eliminated that 10ms jitter. I
don't know if these days setting the low_latency flag (in contexts
where it does work) even has a noticeable effect.
quoted
How can the requirement be for both must-handle-in-minimum-time data
(low_latency) and the-userspace-reader-isn't-reading-fast-enough-
so-its-ok-to-halt-transmission ?
Throttling/unthrottling the sender seems counter to "low latency".
Agreed.
I can _imagine_ a case where an application has a strict limit how
long it will wait until it sees the first byte of the response, but
for some isolated cases (uploading large data logs) that response
might be very large -- large neough that the application may have to
pause while reading the response and rely on flow control to throttle
the upload. But, I can't point to any specific instance of that.
quoted
quoted
_Usually_ applications that require low latency are exchanging short
messages (up to a few hundred bytes, but usually more like a few
dozen). In those cases flow control is not generally needed.
Does it matter?
Driver throttling requires excluding concurrent unthrottle and
calling into the driver (and said driver has relied on sleeping locks
for many kernel versions).
Is that still an issue for drivers where the flow cotrol is handled by
the UART?
Every driver pays the cost to evaluate if the throttling is required,
regardless of how it specifically handles flow control. pty is
now special cased in N_TTY because it forces throttling always on.
quoted
But first I'd like some hard data on whether or not a low latency
mode is even necessary (at least for user-space).
I don't have any hard data, but gut anser is that it is probably no
longer needed.
That's my belief as well.
The problem that existed for the past few years was
that there was a user-space way to set the low_latency flag (it didn't
require root and you could even do it from the command-line with
setserial) -- and doing so annoyed the kernel.
quoted
quoted
Now that HZ is often 1000 and tickless is commonly used, I don't think
the scheduling delay is nearly as much an issue as it used to be. I
haven't gotten any complaints since it was largely rendered useless
several years ago.
Now all my drivers will silently override users if they set the
low_latency flag on a port in situations where it can't be used.
Right. I'd rather not 'fix' something that doesn't really need fixing
(other than to suppress any WARNING caused by low_latency).
Yes. I think what currently needs to be done is to prevent any issues
caused by the user setting the low_latency. [IMO, warnings from the
kernel are an issue, even if the serial port continues to operate
properly.]
Yeah, I'll go fix this.
If somebody has specific latency requirements that aren't being met,
the current performance need to be measured, and it needs to be
decided if a solution is fesable and if the right solution is "fixing"
the low_latency flag so it's allowed in more contexts.
From: One Thousand Gnomes <hidden> Date: 2014-02-19 19:17:37
How can the requirement be for both must-handle-in-minimum-time data
(low_latency) and the-userspace-reader-isn't-reading-fast-enough-
so-its-ok-to-halt-transmission ?
Because low latency is about *turn around* time. There are plenty of
protocols that can flow control, do flow control and want low latency
because they are not windowed. It's not mutually exclusive by any means.
But first I'd like some hard data on whether or not a low latency
mode is even necessary (at least for user-space).
The easy way to simulate the annoying as crap worst cases from dumbass
firmware downloaders and the like is to set up a link between two PCs and
time 2000+ repetitions of
send 64 bytes
wait for a Y
send 64 bytes
wait for a Y
....
and the matching far end being a box running an existing kernel or a PIC
or something doing the responses.
Historically we used to lose about 20mS per cycle which over 2000 got to
be a bit of a PITA.
Low latency goes back to the days of flip buffers, bottom halves an a
100Hz clock. There are certainly better ways to do it now if its needed.
Alan
From: Peter Hurley <hidden> Date: 2014-02-19 20:22:20
On 02/19/2014 02:17 PM, One Thousand Gnomes wrote:
quoted
How can the requirement be for both must-handle-in-minimum-time data
(low_latency) and the-userspace-reader-isn't-reading-fast-enough-
so-its-ok-to-halt-transmission ?
Because low latency is about *turn around* time. There are plenty of
protocols that can flow control, do flow control and want low latency
because they are not windowed. It's not mutually exclusive by any means.
But if it's all about turn around time, how can the situation devolve to
needing throttling in the first place? For N_TTY, throttling only happens
when the read queue is close to overflow (only 128 bytes left in 4k buffer).
If the reader isn't pulling _all_ the data out the instant it's woken,
trying to trim off one worker wakeup (by processing input at interrupt time)
is pointless.
quoted
But first I'd like some hard data on whether or not a low latency
mode is even necessary (at least for user-space).
The easy way to simulate the annoying as crap worst cases from dumbass
firmware downloaders and the like is to set up a link between two PCs and
time 2000+ repetitions of
send 64 bytes
wait for a Y
send 64 bytes
wait for a Y
....
and the matching far end being a box running an existing kernel or a PIC
or something doing the responses.
Well this is easy enough to mock up.
Historically we used to lose about 20mS per cycle which over 2000 got to
be a bit of a PITA.
Low latency goes back to the days of flip buffers, bottom halves an a
100Hz clock. There are certainly better ways to do it now if its needed.
Still, as you've pointed out a couple of times, maybe there's a limited
fastpath that's easy and simple. That's why I was asking about throttling
because it's evaluated even for raw mode.
Regards,
Peter Hurley
From: One Thousand Gnomes <hidden> Date: 2014-02-19 21:43:00
quoted
Because low latency is about *turn around* time. There are plenty of
protocols that can flow control, do flow control and want low latency
because they are not windowed. It's not mutually exclusive by any means.
But if it's all about turn around time, how can the situation devolve to
needing throttling in the first place? For N_TTY, throttling only happens
when the read queue is close to overflow (only 128 bytes left in 4k buffer).
Because if we are on the uplink end of a dumb protocol then our flow ends
up as
< 4K in
read
rest of block
read
block complete
write Y
until at some point sods law decrees that we do a big load of disk
writes, an SMM event steals the processor, firefox garbage collects or
somesuch and we go
4K in
throttle
flow control
It only has to happen *once*, not every time to screw stuff up.
If the reader isn't pulling _all_ the data out the instant it's woken,
trying to trim off one worker wakeup (by processing input at interrupt time)
is pointless.
No - because of the statistical corner cases. The standard Linux is not
remotely hard real time, and even if it were most of the tools involved
are not.
quoted
Low latency goes back to the days of flip buffers, bottom halves an a
100Hz clock. There are certainly better ways to do it now if its needed.
Still, as you've pointed out a couple of times, maybe there's a limited
fastpath that's easy and simple. That's why I was asking about throttling
because it's evaluated even for raw mode.
Even if there isn't the goal of low latency was always 'get stuff to
happen soon' not 'get stuff to happen in the IRQ handler' If you have the
tty processing happening immediately after the IRQ returns when
low_latency is set I'm sure the numbers will be just fine and as good as
historically.
Whether you can always do that I guess depends what happens on really
slow boxes if you don't do any deferral - eg on Geert's Amiga.
Flow control even in raw mode is expected Unix behaviour. However do we
ever need to evaluate it if the tty termios has IXON and CRTSCTS clear ?
Alan
From: Peter Hurley <hidden> Date: 2014-02-20 02:20:04
On 02/19/2014 04:42 PM, One Thousand Gnomes wrote:
It only has to happen *once*, not every time to screw stuff up.
quoted
If the reader isn't pulling _all_ the data out the instant it's woken,
trying to trim off one worker wakeup (by processing input at interrupt time)
is pointless.
No - because of the statistical corner cases. The standard Linux is not
remotely hard real time, and even if it were most of the tools involved
are not.
Ok, so this is still only about "best effort", and really bad
worst case behavior (that the tty core has no control over) is ok.
Going to great lengths to trim one wakeup when nouveau disables interrupts
for 2ms seemed like a waste of time.
quoted
quoted
Low latency goes back to the days of flip buffers, bottom halves an a
100Hz clock. There are certainly better ways to do it now if its needed.
Still, as you've pointed out a couple of times, maybe there's a limited
fastpath that's easy and simple. That's why I was asking about throttling
because it's evaluated even for raw mode.
Even if there isn't the goal of low latency was always 'get stuff to
happen soon' not 'get stuff to happen in the IRQ handler' If you have the
tty processing happening immediately after the IRQ returns when
low_latency is set I'm sure the numbers will be just fine and as good as
historically.
Whether you can always do that I guess depends what happens on really
slow boxes if you don't do any deferral - eg on Geert's Amiga.
Sure.
I think _some_ effort will yield positive results. For example,
in flush_to_ldisc():
if (disc == NULL)
return;
- mutex_lock(&buf->lock);
+ if (!mutex_trylock(&buf->lock))
+ goto deref;
while (1) {
struct tty_buffer *head = buf->head;
.....
mutex_unlock(&buf->lock);
-
+ deref:
tty_ldisc_deref(disc);
}
This change makes flush_to_ldisc() itself safely callable from
interrupt context, and:
1. doesn't lose data (ie., buffers if the ldisc is filling up)
2. automatically picks the optimum handling whether the input worker
is running or not
3. doesn't require more locks to exclude flushing or the input worker
This still leaves fixing termios lock, properly handling throttling
and the exclusion logic for incompatible termios options and low_latency.
[For this change to work properly, I still need to solve a race where
the driver has just updated the commit marker but can't grab the buf
lock because tty_buffer_flush() still has the lock but has already
performed the flush -- in this case, the work needs to be restarted
or something because there's still data in the buffer that needs
processing. There's plenty of possible solutions to this; I'm thinking on
which is the right one].
Flow control even in raw mode is expected Unix behaviour. However do we
ever need to evaluate it if the tty termios has IXON and CRTSCTS clear ?
It doesn't right now because the assumption is that drivers can
have a whole host of reasons why they want to throttle (the pty driver
leaves it on permanently; the vt driver needs unthrottle to support
paste that no one uses even though I fixed it).
Putting aside for a moment the issue of termios safety inside
the throttle and unthrottle driver methods, the exclusion locks here could
be spinlocks if the drivers can be audited/fixed to not sleep here.
Then that just leaves the termios lock, which is a non-trivial problem, and
I'm not convinced RCU will magically fix it.
Regards,
Peter Hurley
From: One Thousand Gnomes <hidden> Date: 2014-02-21 15:40:05
Ok, so this is still only about "best effort", and really bad
worst case behavior (that the tty core has no control over) is ok.
Going to great lengths to trim one wakeup when nouveau disables interrupts
for 2ms seemed like a waste of time.
If it used to work and it doesn't now it's a regression. It's also a
nasty one if you've removed the facility for it.
This change makes flush_to_ldisc() itself safely callable from
interrupt context, and:
1. doesn't lose data (ie., buffers if the ldisc is filling up)
2. automatically picks the optimum handling whether the input worker
is running or not
3. doesn't require more locks to exclude flushing or the input worker
Yep
Putting aside for a moment the issue of termios safety inside
the throttle and unthrottle driver methods, the exclusion locks here could
be spinlocks if the drivers can be audited/fixed to not sleep here.
That was basically insoluble when the lock first went in. We tried with a
spinlock but a lot of USB widgets need to go and chatter with the device
when you do flow control. Flow control is fundamentally ordered but
asynchronous however so if the right fix was to make the USB dongles
queue the work then no harm is done (and the queued flow control
assertion would worst case be no different to a non queued one from a
queued flush_to_ldisc)
Then that just leaves the termios lock, which is a non-trivial problem, and
I'm not convinced RCU will magically fix it.
If you pass a snapshot of the termios state down then I think it does,
but it's still not remotely trivial.
First question though comes before all of this - and that is do we need
low_latency at all any more or is the current scheduling logic now good
enough to do the job anyway.
Alan
From: Peter Hurley <hidden> Date: 2014-02-21 15:58:21
Hi Alan,
On 02/21/2014 10:39 AM, One Thousand Gnomes wrote:
quoted
Ok, so this is still only about "best effort", and really bad
worst case behavior (that the tty core has no control over) is ok.
Going to great lengths to trim one wakeup when nouveau disables interrupts
for 2ms seemed like a waste of time.
If it used to work and it doesn't now it's a regression. It's also a
nasty one if you've removed the facility for it.
I think the consensus is to leave the low_latency facility in, but
remove it's connection to the tty buffers.
If the known-to-be-already-in-non-interrupt-context drivers want,
I can add a different function for executing flush_to_ldisc()
directly. But I don't want to do that without a use-case and test
subject.
quoted
This change makes flush_to_ldisc() itself safely callable from
interrupt context, and:
1. doesn't lose data (ie., buffers if the ldisc is filling up)
2. automatically picks the optimum handling whether the input worker
is running or not
3. doesn't require more locks to exclude flushing or the input worker
Yep
quoted
Putting aside for a moment the issue of termios safety inside
the throttle and unthrottle driver methods, the exclusion locks here could
be spinlocks if the drivers can be audited/fixed to not sleep here.
That was basically insoluble when the lock first went in. We tried with a
spinlock but a lot of USB widgets need to go and chatter with the device
when you do flow control. Flow control is fundamentally ordered but
asynchronous however so if the right fix was to make the USB dongles
queue the work then no harm is done (and the queued flow control
assertion would worst case be no different to a non queued one from a
queued flush_to_ldisc)
Oh. That's not something I want to take on.
quoted
Then that just leaves the termios lock, which is a non-trivial problem, and
I'm not convinced RCU will magically fix it.
If you pass a snapshot of the termios state down then I think it does,
but it's still not remotely trivial.
That was my thought too -- that only dependency injection would work.
Which would require adding that to most, if not all, driver methods, which
seems way too painful.
First question though comes before all of this - and that is do we need
low_latency at all any more or is the current scheduling logic now good
enough to do the job anyway.
Right.
Based on my recent test, I think low_latency doesn't need to be a knob for
the tty core. Drivers can continue to use it to mess with their rx fifo
settings and such like.
I plan on sending Greg a patch to do just that, probably this weekend.
Regards,
Peter Hurley
From: Grant Edwards <hidden> Date: 2014-02-21 16:31:35
On 2014-02-21, Peter Hurley [off-list ref] wrote:
I think the consensus is to leave the low_latency facility in, but
remove it's connection to the tty buffers.
If the known-to-be-already-in-non-interrupt-context drivers want,
I can add a different function for executing flush_to_ldisc()
directly. But I don't want to do that without a use-case and test
subject.
Three of the drivers I maintain have modes where they handle all rx
data in non-interrupt contexts, but I'm not convinced (or even
suspicious) that there would be any noticeable benefit from such a
function. If, at some point in the future, it becomes apparent that
there is "too much latency" in certain cases then perhaps it can be
looked at again -- but I think doing it now is premature optimization.
That said, all things being equal, it would be nice to avoid anything
that would make such an addition impossible in the future.
quoted
First question though comes before all of this - and that is do we need
low_latency at all any more or is the current scheduling logic now good
enough to do the job anyway.
Right.
Based on my recent test, I think low_latency doesn't need to be a
knob for the tty core.
I Agree: there doesn't seem to be any evidence that it's needed by the
tty/ldisc layer.
Drivers can continue to use it to mess with their rx fifo settings
and such like.
Excellent. One of my serial_core drivers still has (in it's default
configuration) 10ms of latency that I can choose to eliminate on a
per-port bases (at the cost of extra CPU cycles) when the low_latency
flag is set.
I plan on sending Greg a patch to do just that, probably this weekend.
Cool. Thanks much for your attention to this.
--
Grant