From: Dave Taht <hidden> Date: 2015-02-01 08:45:45
This convo ended up being cut from netdev due to containing HTML.
I'd really like to get started on finally fixing WiFi starting in April.
---------- Forwarded message ----------
From: "Andrew McGregor" <redacted>
Date: Feb 1, 2015 12:07 AM
Subject: Re: [Cerowrt-devel] Fwd: Throughput regression with `tcp: refine
TSO autosizing`
To: "Avery Pennarun" <redacted>
Cc: "David Reed" <redacted>, "Dave Taht" <redacted>, "Jim
Gettys" [off-list ref], "Tim Shepard" [off-list ref], "Matt
Mathis" [off-list ref], "Jesper Dangaard Brouer" <
jbrouer@redhat.com>, "Jonathan Morton" [off-list ref], "
cerowrt-devel@lists.bufferbloat.net" [off-list ref]
I think I have a good idea how to do it, actually. Or at least, I had a
design sketched out that was kind of plausible.
In essence, we have a good way of controlling queues in the IP stack,
that being fq_codel. But aggregating wifi drivers have to pull all the
traffic straight through that and queue it themselves, with a different
rotation policy and (at present) no AQM.
Well, that can be fixed. We need a transmission and aggregation
scheduler at the wifi layer that can do a per-station round-robin with a
time-based fairness policy, based on airtime. This has to be in the wifi
layer because that's where rate selection policy is, and you can't
calculate airtime until you know the selected rate, the current success
probabilities, and how much aggregation you're going to do within your time
budget. We need to be able to plug in per-flow and per-packet scheduling
and drop policies on top of that, to control scheduling of individual
packets into aggregates.
We could also do the trick of assigning a target drop probability and
telling a Minstrel-like rate controller what drop probability we're after;
then you get the nice property that the transmission scheduler and rate
control are work-conserving and actually get faster when they get congested.
There is also now the mathematics available to improve Minstrel, which is
about 80% of a truly optimal rate controller (interestingly, that math was
published in 2010, years after we did Minstrel; I had the intuition that it
was close to optimal but no way to do a proof).
None of this is rocket science, but it is a pretty deep and complex
refactor of something that isn't the simplest code to start with. That
said, I'd not expect it to be more than around 1k LoC, it's just going to
be a real bear to test.
I think the best path forward is to do proof of concept on the two open
enough drivers we have - the mt76 and ath9k.
A lot of what is needed to make rational decisions is simply not available
from closed firmwares.
I do have the feeling that the people receiving this email ought to be
sufficient to fix the problem, if we can direct enough time to it; for
myself, I have to think quite hard as to whether this should be my 20%
project.
There needs to be focus and direct funding for quite a few people. Felix
for one. It would be nice to acquire half an engineer from each of multiple
companies making wireless products and chips in a linaro like fashion, also.
On Sun, Feb 1, 2015 at 2:06 PM, Avery Pennarun [off-list ref]
wrote:
quoted
I would argue that insofar as the bufferbloat project has made a
difference, it's because there was a very clear message and product:
- here's what sucks when you have bufferbloat
- here's how you can detect it
- here's how you can get rid of it
- by the way, here's which of your competitors are already beating you
at it.
quoted
It turns out you don't need a standards org in order to push any of
the above things. The IEEE exists to make sure things interop at the
MAC/PHY layer. The IETF exists to make sure things interop at the
higher layers. But bufferbloat isn't about interop, it's just a thing
that happens inside gateways, so it's not something you can really
write standards about. It is something you can turn into a
competitive advantage (or disadvantage, if you're a straggler).
...but meanwhile, if we want to fix bufferbloat in wifi, nobody
actually knows how to do it, so we are still at steps 1 and 2.
This is why we're funding Dave to continue work on netperf-wrappers.
I still don't have POs for the other 75% of me this year. It is comforting
to have got such a public confirmation that I have at least this much
floor under me this year, tho.
quoted
Those diagrams of latency under load are pretty convincing. The
diagrams of page load times under different levels of latency are even
more convincing. First, we prove there's a problem and a way to
measure the problem. Then hopefully more people will be interested in
solving it.
Well, a wall of shame might help, I guess. I have a ton of data on a dozen
products that is miserable to see and should be quite embarrassing for the
vendors.
I'd prefer to spend time trying to fix the darn problem more directly, but
I would expect this tool will be of use in tracking improvements.
quoted
On Sat, Jan 31, 2015 at 4:51 PM, [off-list ref] wrote:
quoted
I think we need to create an Internet focused 802.11 working group that
would be to the "OS wireless designers and IEEE 802.11 standards
groups" as
quoted
quoted
the WHATML group was to W3C.
W3C was clueless about the real world at the point WHATML was
created. And
quoted
quoted
WHATML was a "revenge of the real" against W3C - advancing a wide
variety of
quoted
quoted
important practical innovations rather than attending endless standards
meetings with people who were not focused on solving actually important
problems.
It took a bunch of work to get WHATML going, and it offended W3C, who
became
quoted
quoted
unhelpful. But the approach actually worked - we now have a Web that
really
quoted
quoted
uses browser-side expressivity and that would never have happened if
W3C
quoted
quoted
were left to its own devices.
The WiFi consortium was an attempt to wrest control of pragmatic
direction
quoted
quoted
from 802.11 and the proprietary-divergence folks at Qualcomm, Broadcom,
Cisco, etc. But it failed, because it became thieves on a raft, more
focused on picking each others' pockets than on actually addressing
the big
quoted
quoted
issues.
Jim has seen this play out in the Linux community around X. Though
there
quoted
quoted
are lots of interests who would benefit by moving the engineering ball
forward, everyone resists action because it means giving up the chance
at
quoted
quoted
dominance, and the central group is far too weak to do anything beyond
adjudicating the worst battles.
When I say "we" I definitely include myself (though my time is limited
due
quoted
quoted
to other commitments and the need to support my family), but I would
only
quoted
quoted
play with people who actually are committed to making stuff happen -
which
quoted
quoted
includes raising hell with the vendors if need be, but also effective
engineering steps that can achieve quick adoption.
Sadly, and I think it is manageable at the moment, there are moves out
there
quoted
quoted
being made to get the FCC to "protect" WiFi from "interference". The
current one was Marriott, who requested the FCC for a rule to make it
legal
quoted
quoted
to disrupt and block use of WiFi in people's rooms in their hotels,
except
quoted
quoted
with their access points. This also needs some technical defense. I
believe any issues with WiFi performance in actual Marriott hotels are
due
quoted
quoted
to bufferbloat in their hotel-wide systems, just as the issues with
GoGo are
quoted
quoted
the same. But it's possible that queueing problems in their own WiFi
gear
quoted
quoted
are bad as well.
I mention this because it is related, and to the layperson, or
non-radio-knowledgeable executive, indistinguishable. It will take
away the
quoted
quoted
incentive to actually fix the 802.11 implementations to be better
performing, making the problem seem to be a "management" issue that
can be
quoted
quoted
solved by making WiFi less interoperable and less flexible by rules,
rather
quoted
quoted
than by engineering.
However, solving the problems of hotspot networks and hotel networks
are
quoted
quoted
definitely "real world" issues, and quite along the same lines you
mention,
quoted
quoted
Dave. FQ is almost certainly a big deal both in WiFi and in the
distribution networks behind WiFi. Co-existence is also a big deal
(RTS/CTS-like mechanisms can go a long way to remediate hidden-terminal
disruption of the basic protocols). Roaming and scaling need work as
well.
quoted
quoted
It would even be a good thing to invent pragmatic ways to provide "low
rate"
quoted
quoted
subnets and "high rate" subnets that can coexist, so that
compatibility with
quoted
quoted
ancient "b" networks need not be maintained on all nets, at great cost
-
quoted
quoted
just send beacons at a high rate, so that the "b" NICs can't see
them....
quoted
quoted
but you need pragmatic stack implementations.
But the engineering is not the only challenge. The other challenge is
to
quoted
quoted
take the initiative and get stuff deployed. In the case of
bufferbloat, the
quoted
quoted
grade currently is a "D" for deployments, maybe a "D-". Beautiful
technical
quoted
quoted
work, but the economic/business/political side of things has been poor.
Look at how slow IETF has been to achieve anything (the perfect is
truly the
quoted
quoted
enemy of the good, and Dave Clark's "rough consensus and working code"
has
quoted
quoted
been replaced by technocratic malaise, and what appears to me to be a
class
quoted
quoted
of people who love traveling the world to a floating cocktail party
without
quoted
quoted
getting anything important done).
The problem with communications is that you can't just ship a product
with a
quoted
quoted
new "feature", because the innovation only works if widely adopted.
Since
quoted
quoted
there is no "Linux Desktop" (and Linus hates the idea, to a large
extent)
quoted
quoted
Linux can't be the sole carrier of the idea. You pretty much need iOS
and
quoted
quoted
Android both to buy in or to provide a path for easy third-party
upgrades.
quoted
quoted
How do you do that? Well, that's where the WHATML-type approach is
necessary.
I don't know if this can be achieved, and there are lots of details to
be
quoted
quoted
worked out. But I'll play.
I'll play.
quoted
quoted
On Saturday, January 31, 2015 4:05pm, "Dave Taht" [off-list ref]
said:
I would like to have somehow assembled all the focused resources to
make a
quoted
quoted
go at fixing wifi, or at least having a f2f with a bunch of people in
the
quoted
quoted
late march timeframe. This message of mine to linux-wireless bounced
for
quoted
quoted
some reason and I am off to log out for 10 days, so...
see relevant netdev thread also for ore details.
---------- Forwarded message ----------
From: Dave Taht <redacted>
Date: Sat, Jan 31, 2015 at 12:29 PM
Subject: Re: Throughput regression with `tcp: refine TSO autosizing`
To: Arend van Spriel <redacted>
Cc: linux-wireless <redacted>, Michal Kazior
[off-list ref], Eyal Perry [off-list ref],
Network
quoted
quoted
Development [off-list ref], Eric Dumazet <
eric.dumazet@gmail.com>
quoted
quoted
The wifi industry as a whole has vastly bigger problems than achieving
1500Mbits in a faraday cage on a single flow.
I encourage you to try tests in netperf-wrapper that explicitly test
for
quoted
quoted
latency under load, and in particular, the RTT_FAIR tests against 4 or
more
quoted
quoted
stations on a single wifi AP. You will find the results very
depressing.
quoted
quoted
Similarly, on your previous test series, a latency figure would have
been
quoted
quoted
nice to have. I just did a talk at nznog, where I tested the local
coffee shops, cities... really, anywhere outside a faraday cage!
I am not attending netconf - I was unable to raise funds to go, and the
program committee wanted something "new",
instead of the preso I gave the IEEE 802.11 working group back in
)
I was very pleased with the results of that talk - the day after I
gave it,
quoted
quoted
the phrase "test for latency" showed up in a bunch of 802.11ax (the
next
quoted
quoted
generation after ac) documents. :) Still, we are stuck with the train
wreck
quoted
quoted
that is 802.11ac glommed on top of 802.11n, glommed on top of 802.11g,
in
quoted
quoted
terms of queue management, terrible uses of airtime, rate control and
other
quoted
quoted
stuff. Aruba and Meraki, in particular took a big interest in what I'd
outlined in the preso above (we have a half dozen less well baked
ideas -
quoted
quoted
that's just the easy stuff that can be done to improve wifi). I gave a
followup at meraki but I don't think that's online.
Felix (nbd) is on vacation right now, as I am I. In fact I am going
somewhere for a week totally lacking internet access.
Presently the plan, with what budget (none) we have and time (very
little)
quoted
quoted
we have is to produce a pair of proof of concept implementations for
per tid
quoted
quoted
queuing (see relevant commit by nbd), leveraging the new minstrel
stats,
quoted
quoted
the new minstrel-blues stuff, and an aggregation aware codel with a
calculated target based on the most recently active stations, and a
bunch of
quoted
quoted
the other stuff outlined above at IEEE.
It is my hope that this will start to provide accurate back pressure
(or
quoted
quoted
sufficient lack thereof for TSQ), to also improve throughput while
still
quoted
quoted
retaining low latency. But it is a certainty that we will run into more
cross layer issues that will be a pita to resolve.
If we can put together a meet up around or during ELC in california in
march?
I am really not terribly optimistic on anything other than the 2
chipsets we
quoted
quoted
can hack on (ath9k, mt76). Negotiations to get qualcomm to open up
their
quoted
quoted
ath10k firmware have thus far failed, nor has a ath10k-lite got
anywhere.
quoted
quoted
Perhaps broadcom would be willing to open up their firmware
sufficiently to
quoted
quoted
build in a better API?
A bit more below.
On Jan 30, 2015 5:59 AM, "Arend van Spriel" [off-list ref] wrote:
quoted
On 01/30/15 14:19, Eric Dumazet wrote:
quoted
On Fri, 2015-01-30 at 11:29 +0100, Arend van Spriel wrote:
quoted
Hi Eric,
Your suggestions are still based on the fact that you consider
wireless
quoted
quoted
quoted
quoted
quoted
networking to be similar to ethernet, but as Michal indicated there
are
quoted
quoted
quoted
quoted
quoted
some fundamental differences starting with CSMA/CD versus CSMA/CA.
Also
quoted
quoted
quoted
quoted
quoted
the medium conditions are far from comparable.
The analogy i now use for it is that switched ethernet is generally
your
quoted
quoted
classic "dumbbell"
topology. Wifi is more like a "taxi-stand" topology. If you think
about how
quoted
quoted
people
queue up at a taxi stand (and sometimes agree to share a ride), the
inter
quoted
quoted
arrival
and departure times of a taxi stand make for a better mental model.
Admittedly, I seem to spend a lot of time, waiting for taxies, thinking
about
wifi.
quoted
quoted
There is no shielding so
quoted
it needs to deal with interference and dynamically drops the link
rate
quoted
quoted
quoted
quoted
quoted
so transmission of packets can take several milliseconds. Then with
11n
quoted
quoted
quoted
quoted
quoted
they came up with aggregation with sends up to 64 packets in a
single
quoted
quoted
quoted
quoted
quoted
transmit over the air at worst case 6.5 Mbps (if I am not
mistaken). The
quoted
quoted
quoted
quoted
quoted
parameter value for tcp_limit_output_bytes of 131072 means that it
allows queuing for about 1ms on a 1Gbps link, but I hope you can see
this is not realistic for dealing with all variances of the wireless
medium/standard. I suggested this as topic for the wireless
workshop in
quoted
quoted
quoted
quoted
quoted
Otawa [1], but I can not attend there. Still hope that there will be
some discussions to get more awareness.
I have sometimes hoped that TSQ could be made more a function of the
number of active flows exiting an interface, but eric tells me that's
impossible.
This is possibly another case where TSQ could use to be a callback
function...
but frankly I care not a whit about maximizing single flow tcp
throughput on
quoted
quoted
wifi
in a faraday cage.
quoted
quoted
Ever heard about bufferbloat ?
Sure. I am trying to get awareness about that in our wireless
driver/firmware development teams. So bear with me.
quoted
Have you read my suggestions and tried them ?
You can adjust the limit per flow to pretty much you want. If you
need
quoted
quoted
quoted
quoted
64 packets, just do the math. If in 2018 you need 128 packets, do the
math again.
I am very well aware that wireless wants aggregation, thank you.
I note that a lot of people testing this are getting it backwards.
Usually
quoted
quoted
it is the AP that is sending lots and lots of big packets, where the
return
quoted
quoted
path is predominately acks from the station.
I am not a huge fan of stretch acks, but certainly a little bit of
thinning
quoted
quoted
doesn't bother me on the return path there.
Going the other way, particularly in a wifi world that insists on
treating
quoted
quoted
every packet as sacred (which I don't agree with at all), thinning
acks can
quoted
quoted
help, but single stream throughput is of interest only on benchmarks,
FQing
quoted
quoted
as much as possible all the flows destined the station in each
aggregate
quoted
quoted
masks loss and reduces the need to protect everything so much.
quoted
Sorry if I offended you. I was just giving these as example combined
with
quoted
quoted
quoted
effective rate usable on the medium to say that the bandwidth is more
dynamic in wireless and as such need dynamic change of queue depth.
Now this
quoted
quoted
quoted
can be done by making the fraction size as used in your suggestion
adaptive
quoted
quoted
quoted
to these conditions.
Well... see above. Maybe this technique will do more of the right
thing,
quoted
quoted
but... go test.
quoted
quoted
131072 bytes of queue on 40Gbit is not 1ms, but 26 usec of queueing,
and
quoted
quoted
quoted
quoted
we get line rate nevertheless.
I was saying it was about 1ms on *1Gbit* as the wireless TCP rates are
moving into that direction in 11ac.
quoted
We need this level of shallow queues (BQL, TSQ), to get very precise
rtt
quoted
quoted
quoted
quoted
estimations so that TCP has good entropy for its pacing, even in the
50
quoted
quoted
quoted
quoted
usec rtt ranges.
If we allowed 1ms of queueing, then a 40Gbit flow would queue 5
MBytes.
quoted
quoted
quoted
quoted
This was terrible, because it increased cwnd and all sender queues to
insane levels.
Indeed and that is what we would like to address in our wireless
drivers.
quoted
quoted
quoted
I will setup some experiments using the fraction sizing and post my
findings. Again sorry if I offended you.
You really, really, really need to test at rates below 50mbit and with
other
quoted
quoted
stations, also while doing this. It's not going to be a linear curve.
quoted
Regards,
Arend
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Jonathan Morton <hidden> Date: 2015-02-01 10:47:23
Since this is going to be a big job, it's worth prioritising parts of it
appropriately.
Minstrel is probably already the single best feature of the Linux Wi-Fi
stack. AFAIK it still outperforms any other rate selector we know about. So
I don't consider improving it further to be a high priority, although that
trick of using it as a sneaky random packet loss inducer is intriguing.
Much more important and urgent is getting some form of functioning SQM
closer to the hardware, where the information is. I don't think we need to
get super fancy here to do some real good, in the same way that PIE is a
major improvement over drop-tail. I'd settle for a variant of fq_codel that
gets and uses information about whether the current packet request might be
aggregated with the previous packet provided, and adjusts its choice of
packet accordingly.
At the same time, models would undoubtedly be useful to help test and
repeatably demonstrate the advantages of both simple and more sophisticated
solutions. Ns3 allows laying out a reasonably complex radio environment,
which is great for this. To counter the prevalence of one-station Faraday
cage tests in the industry, the simulated environments should represent
realistic, challenging use cases:
1) the family home, with half a dozen client devices competing with several
interference sources (Bluetooth, RC toys, microwave oven, etc). This is a
relatively easy environment, representing the expected environment for
consumer equipment.
2) the apartment block, with fewer clients per AP but lots of APs
distributed throughout a large building. Walls and floors may provide
valuable attenuation here - unless you're in Japan, where they can be
notoriously thin.
3) the railway carriage, consisting of eighty passengers in a 20x3 m space,
and roughly the same number of client devices. The uplink is 3G based and
has some inherent latency. Add some Bluetooth for flavour, stir gently.
This one is rather challenging, but there is scope to optimise AP antenna
placement, and to scale the test down slightly by reducing seat occupancy.
4) the jumbo jet, consisting of several hundred passengers crammed in like
sardines. The uplink has satellite latencies built in. Good luck.
5) the business hotel. Multiple APs will be needed to provide adequate
coverage for this environment, which should encompass the rooms as well as
lounge, conference and dining areas. Some visitors may bring their own APs,
and the system must be able to cope with this without seriously degrading
performance.
6) the trade conference. A large arena filled with thousands of people.
Multiple APs required. Good luck.
I also feel that ultimately we're going to have to get industry on board.
Not just passively letting us play around as with ath9k, but actively
taking note of our findings and implementing at least a few of our ideas
themselves. Of course, tools, models and real-world results are likely to
make that easier.
- Jonathan Morton
Just to clarify, managing queueing in a single access point WiFi network is only a small part of the problem of fixing the rapidly degrading performance of WiFi based systems. Similarly, mesh routing is only a small part of the problem with the scalability of cooperative meshes based on the WiFi MAC.
So I don't disagree with work on queue management (which is not good in any commercial product).
But Dave T has done some great talks on "fixing WiFi" that don't have to do with queueing very much at all.
For example, rate selection for various packets is terrible. When you have nearly 1000:1 ratios of transmission rates and codes that are not backward compatible, there's a huge opportunity for improvement. Similarly, choice of frequency bandwidth and center frequency at each station offers huge opportunities for practical scalability of systems. Also, as we noted earlier, "handoff" from one next hop to another is a huge problem with performance in practical deployments (a factor of 10x at least, just in that).
Propagation information is not used at all when 802.11 systems share a channel, even in single AP deployments, yet all stations can measure propagation quite accurately in their hardware.
Finally, Listen-before-talk is highly wasteful for two reasons: 1) any random radio noise from other sources unnecessarily degrades communications (and in the 5.8 MHz band, the rule about "radar avoidance" requires treating very low level noise as a "signal to shut the net down by law", but there is a loophole if you can tell that it's not actually "radar" (the technique requires two or more stations to measure the same noise event, and if the power is significantly different - more than a few dB - then it can't possibly be due to a distant transmitter, and therefore can be ignored). 2) the transmitter cannot tell when the intended receiver will be perfectly able to decode the signal without interference with the station it hears (this second point is actually proven in theory in a paper by Jon Peha that argued against trivial "etiquettes" as a mechanism for sharing among uncooperative and non-interoperable stations).
Dave T has discussed more, as have I in other venues.
The reason no one is making progress on any of these particular issues is that there is no coordination at the "systems level" around creating rising tides that lift all boats in the WiFi-ish space. It's all about ripping the competition by creating stuff that can sell better than the other guys' stuff, and avoiding cooperation at all costs.
I agree that, to the extent that managing queues in a single box or a single operating system doesn't require cooperation, it's much easier to get such things into the market. That's why CeroWRT has been as effective as it has been. But has Microsoft done anything at all about it? Do the better ECN signals that can arise from good queue management get used by the TCP endpoints, or for that matter UDP-based protocol endpoints?
But the big wins in making WiFi better are going begging. As WiFi becomes more closed, as it will as the major Internet Access Providers and Gadget builders (Google, Apple) start excluding innovators in wireless from the market by closed, proprietary solutions, the problem WILL get worse. You won't be able to fix those problems at all. If you have a solution you will have to convince the oligopoly to even bother trying it.
So, let me reiterate. The problem is not just "getting Minstrel adopted", though I have nothing against that as a subgoal. The problem is to find good systems-level answers, and to find a strategy to deliver those answers to a WiFi ecology that spans the planet, and where the marketing value-story focuses on things one can measure between two stations in a Faraday cage, and never on any systems-level issues.
I personally think that things like promoting semi-closed, essentially proprietary ESSID-based bridged distribution systems as "good ideas" are counterproductive to this goal. But that's perhaps too radical for this crowd. It reminds me of Cisco's attempt to create a proprietary Internet technology with IOS, which fortunately was not the success Cisco hoped for, or Juniper would not have existed. Maybe IOS would have been a fine standard, but it would have killed the evolution of the Internet as we know it.
On Sunday, February 1, 2015 5:47am, "Jonathan Morton" [off-list ref] said:
Since this is going to be a big job, it's worth prioritising parts of it appropriately.
Minstrel is probably already the single best feature of the Linux Wi-Fi stack. AFAIK it still outperforms any other rate selector we know about. So I don't consider improving it further to be a high priority, although that trick of using it as a sneaky random packet loss inducer is intriguing.
Much more important and urgent is getting some form of functioning SQM closer to the hardware, where the information is. I don't think we need to get super fancy here to do some real good, in the same way that PIE is a major improvement over drop-tail. I'd settle for a variant of fq_codel that gets and uses information about whether the current packet request might be aggregated with the previous packet provided, and adjusts its choice of packet accordingly.
At the same time, models would undoubtedly be useful to help test and repeatably demonstrate the advantages of both simple and more sophisticated solutions. Ns3 allows laying out a reasonably complex radio environment, which is great for this. To counter the prevalence of one-station Faraday cage tests in the industry, the simulated environments should represent realistic, challenging use cases:
1) the family home, with half a dozen client devices competing with several interference sources (Bluetooth, RC toys, microwave oven, etc). This is a relatively easy environment, representing the expected environment for consumer equipment.
2) the apartment block, with fewer clients per AP but lots of APs distributed throughout a large building. Walls and floors may provide valuable attenuation here - unless you're in Japan, where they can be notoriously thin.
3) the railway carriage, consisting of eighty passengers in a 20x3 m space, and roughly the same number of client devices. The uplink is 3G based and has some inherent latency. Add some Bluetooth for flavour, stir gently. This one is rather challenging, but there is scope to optimise AP antenna placement, and to scale the test down slightly by reducing seat occupancy.
4) the jumbo jet, consisting of several hundred passengers crammed in like sardines. The uplink has satellite latencies built in. Good luck.
5) the business hotel. Multiple APs will be needed to provide adequate coverage for this environment, which should encompass the rooms as well as lounge, conference and dining areas. Some visitors may bring their own APs, and the system must be able to cope with this without seriously degrading performance.
6) the trade conference. A large arena filled with thousands of people. Multiple APs required. Good luck.
I also feel that ultimately we're going to have to get industry on board. Not just passively letting us play around as with ath9k, but actively taking note of our findings and implementing at least a few of our ideas themselves. Of course, tools, models and real-world results are likely to make that easier.
- Jonathan Morton
From: Andrew McGregor <hidden> Date: 2015-02-01 23:34:10
So far as I'm concerned, Minstrel is as widely adopted as we could want; it
could go further, but it doesn't need any help with that. It could be
better, however, and that's a fairly straightforward change. One of the
problems with rate selection is that the standard itself has a bit to say
on what rates to use for what packets, and what it says is braindead. So
if you want to be standards compliant you are also leaving quite a bit of
potential performance on the table.
I missed one item in my list of potential improvements: the most braindead
thing 802.11 has to say about rates is that broadcast and multicast packets
should be sent at 'the lowest basic rate in the current supported rate
set', which is really wasteful. There are a couple of ways of dealing with
this: one, ignore the standard and pick the rate that is most likely to get
the frame to as many neighbours as possible (by a scan of the Minstrel
tables). Or two, fan it out as unicast, which might well take less airtime
(due to aggregation) as well as being much more likely to be delivered,
since you get ACKs and retries by doing that.
As for the industry... sure, there are those forces, but then again there's
a non-trivial amount of influence in this audience.
On Mon, Feb 2, 2015 at 1:43 AM, [off-list ref] wrote:
Just to clarify, managing queueing in a single access point WiFi network
is only a small part of the problem of fixing the rapidly degrading
performance of WiFi based systems. Similarly, mesh routing is only a small
part of the problem with the scalability of cooperative meshes based on the
WiFi MAC.
So I don't disagree with work on queue management (which is not good in
any commercial product).
But Dave T has done some great talks on "fixing WiFi" that don't have to
do with queueing very much at all.
For example, rate selection for various packets is terrible. When you
have nearly 1000:1 ratios of transmission rates and codes that are not
backward compatible, there's a huge opportunity for improvement.
Similarly, choice of frequency bandwidth and center frequency at each
station offers huge opportunities for practical scalability of systems.
Also, as we noted earlier, "handoff" from one next hop to another is a huge
problem with performance in practical deployments (a factor of 10x at
least, just in that).
Propagation information is not used at all when 802.11 systems share a
channel, even in single AP deployments, yet all stations can measure
propagation quite accurately in their hardware.
Finally, Listen-before-talk is highly wasteful for two reasons: 1) any
random radio noise from other sources unnecessarily degrades communications
(and in the 5.8 MHz band, the rule about "radar avoidance" requires
treating very low level noise as a "signal to shut the net down by law",
but there is a loophole if you can tell that it's not actually "radar" (the
technique requires two or more stations to measure the same noise event,
and if the power is significantly different - more than a few dB - then it
can't possibly be due to a distant transmitter, and therefore can be
ignored). 2) the transmitter cannot tell when the intended receiver will be
perfectly able to decode the signal without interference with the station
it hears (this second point is actually proven in theory in a paper by Jon
Peha that argued against trivial "etiquettes" as a mechanism for sharing
among uncooperative and non-interoperable stations).
Dave T has discussed more, as have I in other venues.
The reason no one is making progress on any of these particular issues is
that there is no coordination at the "systems level" around creating rising
tides that lift all boats in the WiFi-ish space. It's all about ripping
the competition by creating stuff that can sell better than the other guys'
stuff, and avoiding cooperation at all costs.
I agree that, to the extent that managing queues in a single box or a
single operating system doesn't require cooperation, it's much easier to
get such things into the market. That's why CeroWRT has been as effective
as it has been. But has Microsoft done anything at all about it? Do the
better ECN signals that can arise from good queue management get used by
the TCP endpoints, or for that matter UDP-based protocol endpoints?
But the big wins in making WiFi better are going begging. As WiFi becomes
more closed, as it will as the major Internet Access Providers and Gadget
builders (Google, Apple) start excluding innovators in wireless from the
market by closed, proprietary solutions, the problem WILL get worse. You
won't be able to fix those problems at all. If you have a solution you
will have to convince the oligopoly to even bother trying it.
So, let me reiterate. The problem is not just "getting Minstrel adopted",
though I have nothing against that as a subgoal. The problem is to find
good systems-level answers, and to find a strategy to deliver those answers
to a WiFi ecology that spans the planet, and where the marketing
value-story focuses on things one can measure between two stations in a
Faraday cage, and never on any systems-level issues.
I personally think that things like promoting semi-closed, essentially
proprietary ESSID-based bridged distribution systems as "good ideas" are
counterproductive to this goal. But that's perhaps too radical for this
crowd. It reminds me of Cisco's attempt to create a proprietary Internet
technology with IOS, which fortunately was not the success Cisco hoped for,
or Juniper would not have existed. Maybe IOS would have been a fine
standard, but it would have killed the evolution of the Internet as we know
it.
On Sunday, February 1, 2015 5:47am, "Jonathan Morton" <
chromatix99@gmail.com> said:
Since this is going to be a big job, it's worth prioritising parts of it
appropriately.
Minstrel is probably already the single best feature of the Linux Wi-Fi
stack. AFAIK it still outperforms any other rate selector we know about. So
I don't consider improving it further to be a high priority, although that
trick of using it as a sneaky random packet loss inducer is intriguing.
Much more important and urgent is getting some form of functioning SQM
closer to the hardware, where the information is. I don't think we need to
get super fancy here to do some real good, in the same way that PIE is a
major improvement over drop-tail. I'd settle for a variant of fq_codel that
gets and uses information about whether the current packet request might be
aggregated with the previous packet provided, and adjusts its choice of
packet accordingly.
At the same time, models would undoubtedly be useful to help test and
repeatably demonstrate the advantages of both simple and more sophisticated
solutions. Ns3 allows laying out a reasonably complex radio environment,
which is great for this. To counter the prevalence of one-station Faraday
cage tests in the industry, the simulated environments should represent
realistic, challenging use cases:
1) the family home, with half a dozen client devices competing with
several interference sources (Bluetooth, RC toys, microwave oven, etc).
This is a relatively easy environment, representing the expected
environment for consumer equipment.
2) the apartment block, with fewer clients per AP but lots of APs
distributed throughout a large building. Walls and floors may provide
valuable attenuation here - unless you're in Japan, where they can be
notoriously thin.
3) the railway carriage, consisting of eighty passengers in a 20x3 m
space, and roughly the same number of client devices. The uplink is 3G
based and has some inherent latency. Add some Bluetooth for flavour, stir
gently. This one is rather challenging, but there is scope to optimise AP
antenna placement, and to scale the test down slightly by reducing seat
occupancy.
4) the jumbo jet, consisting of several hundred passengers crammed in like
sardines. The uplink has satellite latencies built in. Good luck.
5) the business hotel. Multiple APs will be needed to provide adequate
coverage for this environment, which should encompass the rooms as well as
lounge, conference and dining areas. Some visitors may bring their own APs,
and the system must be able to cope with this without seriously degrading
performance.
6) the trade conference. A large arena filled with thousands of people.
Multiple APs required. Good luck.
I also feel that ultimately we're going to have to get industry on board.
Not just passively letting us play around as with ath9k, but actively
taking note of our findings and implementing at least a few of our ideas
themselves. Of course, tools, models and real-world results are likely to
make that easier.
- Jonathan Morton
On Sun, Feb 1, 2015 at 9:43 AM, [off-list ref] wrote:
Just to clarify, managing queueing in a single access point WiFi network is
only a small part of the problem of fixing the rapidly degrading performance
of WiFi based systems.
Can you explain what you mean by "rapidly degrading?" The performance
in odd situations is certainly not inspirational, but I haven't
noticed it getting worse over time.
Similarly, mesh routing is only a small part of the
problem with the scalability of cooperative meshes based on the WiFi MAC.
That's certainly true. Not to say the mesh routing algorithms are
much good either.
Also, as we noted
earlier, "handoff" from one next hop to another is a huge problem with
performance in practical deployments (a factor of 10x at least, just in
that).
While there is definitely some work to be done in handoff, it seems
like there are some find implementations of this already in existence.
Several brands of "enterprise access point" setups seem to do well at
this. It would be nice if they interoperated, I guess.
The fact that there's no open source version of this kind of handoff
feature bugs me, but we are working on it here and the work is all
planned to be open source, for example: (very early version)
https://gfiber.googlesource.com/vendor/google/platform/+/master/waveguide/
Propagation information is not used at all when 802.11 systems share a
channel, even in single AP deployments, yet all stations can measure
propagation quite accurately in their hardware.
802.11k seems to provide for sharing this information. But I'm not
clear what I should use it for. :)
Finally, Listen-before-talk is highly wasteful for two reasons: 1) any
random radio noise from other sources unnecessarily degrades communications [...]
2) the transmitter cannot tell when the intended receiver will be perfectly
able to decode the signal without interference with the station it hears
(this second point is actually proven in theory in a paper by Jon Peha that
argued against trivial "etiquettes" as a mechanism for sharing among
uncooperative and non-interoperable stations).
I've thought quite a bit about your point #2 above, but I don't know
which direction to pursue. The idea is that sometimes "just shout
over the background noise" is a globally optimal solution, right? The
question seems to be to figure out when that is true and when it
isn't.
I agree that, to the extent that managing queues in a single box or a single
operating system doesn't require cooperation, it's much easier to get such
things into the market. That's why CeroWRT has been as effective as it has
been. But has Microsoft done anything at all about it? Do the better ECN
signals that can arise from good queue management get used by the TCP
endpoints, or for that matter UDP-based protocol endpoints?
If we don't know the answer to the questions, then that is itself the
problem. It's a lot easier to say, hey, ChromeOS and MacOS have good
network performance but Microsoft has bad network performance, if it's
true and we have good reproducible tests to demonstrate that.
The reason no one is making progress on any of these particular issues is
that there is no coordination at the "systems level" around creating rising
tides that lift all boats in the WiFi-ish space. It's all about ripping the
competition by creating stuff that can sell better than the other guys'
stuff, and avoiding cooperation at all costs.
[...]
But the big wins in making WiFi better are going begging. As WiFi becomes
more closed, as it will as the major Internet Access Providers and Gadget
builders (Google, Apple) start excluding innovators in wireless from the
market by closed, proprietary solutions, the problem WILL get worse. You
won't be able to fix those problems at all. If you have a solution you will
have to convince the oligopoly to even bother trying it.
As someone who works at Google Fiber (which is both a gadget maker and
an ISP) and who pushes all day long for our wifi stuff to be open
source, I'm slightly offended to be lumped in with other vendors in
your story :) I think the ChromeOS team (which insists on only open
source wifi drivers in all chromebooks) would feel similarly. We are
lucky to have defined our competitive advantage as something other
than short-lived slight improvements in wifi that will soon be
wastefully duplicated by everyone else.
That said, I see what you mean about the general state of the
industry. The way to fix it is the way Linux always fixes it: make
the open source version so much better that building a proprietary
one, just to gather a small incremental advantage, is a huge waste of
time and effort. Work on minstrel and fq_codel go really far here.
I personally think that things like promoting semi-closed, essentially
proprietary ESSID-based bridged distribution systems as "good ideas" are
counterproductive to this goal. But that's perhaps too radical for this
crowd.
Not sure what you mean here. ESSID-based distribution systems seem
pretty well defined to me. The only proprietary part is the
decision-making process for assisted roaming (ie. the "inter-AP
protocol") which is only an optional performance optimization. There
really should be an open source version of this, and I'm in fact
feebly attempting to build one, but I don't feel like the world is
falling apart through not having it. You can build a bridged
multi-BSS ESSID today with plain out-of-the-box hostapd.
Have fun,
Avery
From: David Lang <hidden> Date: 2015-02-02 07:07:23
On Sun, 1 Feb 2015, Avery Pennarun wrote:
On Sun, Feb 1, 2015 at 9:43 AM, [off-list ref] wrote:
quoted
I personally think that things like promoting semi-closed, essentially
proprietary ESSID-based bridged distribution systems as "good ideas" are
counterproductive to this goal. But that's perhaps too radical for this
crowd.
Not sure what you mean here. ESSID-based distribution systems seem
pretty well defined to me. The only proprietary part is the
decision-making process for assisted roaming (ie. the "inter-AP
protocol") which is only an optional performance optimization. There
really should be an open source version of this, and I'm in fact
feebly attempting to build one, but I don't feel like the world is
falling apart through not having it. You can build a bridged
multi-BSS ESSID today with plain out-of-the-box hostapd.
I will be running a fully opensource bridged ESSID system at SCaLE this month. last year we had ~2500 people and devices with ~50 APs deployed, and it worked well. The only problem was that I needed to deploy a few more APs to cover some of the hallway areas more reliably.
There are tricks that the commercial systems pull that I can't currently duplicate with opensource tools. But as Avery says, they are optimizations, not something required for successful operation. It would be nice to get the assisted roaming portion available. But it's not required.
David Lang