From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2011-01-27 20:06:06
On Thu, Jan 27, 2011 at 11:45:47AM -0800, Shirley Ma wrote:
On Thu, 2011-01-27 at 21:31 +0200, Michael S. Tsirkin wrote:
quoted
Well slowing down the guest does not sound hard - for example we can
request guest notifications, or send extra interrupts :)
A slightly more sophisticated thing to try is to
poll the vq a bit more aggressively.
For example if we handled some requests and now tx vq is empty,
reschedule and yeild. Worth a try?
I used dropping packets in high level to slow down TX.
I am still
thinking what's the right the approach here.
Interesting. Could this is be a variant of the now famuous bufferbloat then?
I guess we could drop some packets if we see we are not keeping up. For
example if we see that the ring is > X% full, we could quickly complete
Y% without transmitting packets on. Or maybe we should drop some bytes
not packets.
Requesting guest notification and extra interrupts is what we want to
avoid to reduce VM exits for saving CPUs. I don't think it's good.
Yes but how do you explain regression?
One simple theory is that guest net stack became faster
and so the host can't keep up.
By polling the vq a bit more aggressively, you meant vhost, right?
Shirley
From: Shirley Ma <hidden> Date: 2011-01-27 20:15:33
On Thu, 2011-01-27 at 22:05 +0200, Michael S. Tsirkin wrote:
Interesting. Could this is be a variant of the now famuous bufferbloat
then?
I guess we could drop some packets if we see we are not keeping up.
For
example if we see that the ring is > X% full, we could quickly
complete
Y% without transmitting packets on. Or maybe we should drop some bytes
not packets.
It's worth to try to figure out what's the best approach. I will make a
patch.
quoted
Requesting guest notification and extra interrupts is what we want
to
quoted
avoid to reduce VM exits for saving CPUs. I don't think it's good.
Yes but how do you explain regression?
One simple theory is that guest net stack became faster
and so the host can't keep up.
Yes, that's what I think here. Some qdisc code has been changed
recently.
quoted
By polling the vq a bit more aggressively, you meant vhost, right?
Shirley
Yes.
I had a similar patch before, I can modify it and test it out.
Shirley
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2011-01-28 12:16:16
On Thu, Jan 27, 2011 at 01:30:38PM -0800, Shirley Ma wrote:
On Thu, 2011-01-27 at 13:02 -0800, David Miller wrote:
quoted
quoted
Interesting. Could this is be a variant of the now famuous
bufferbloat then?
Sigh, bufferbloat is the new global warming... :-/
Yep, some places become colder, some other places become warmer; Same as
BW results, sometimes faster, sometimes slower. :)
Shirley
OK, so thinking about it more, maybe the issue is this:
tx becomes full. We process one request and interrupt the guest,
then it adds one request and the queue is full again.
Maybe the following will help it stabilize?
By itself it does nothing, but if you set
all the parameters to a huge value we will
only interrupt when we see an empty ring.
Which might be too much: pls try other values
in the middle: e.g. make bufs half the ring,
or bytes some small value, or packets some
small value etc.
Warning: completely untested.
@@ -127,6 +134,9 @@ static void handle_tx(struct vhost_net *net)interr,wmem;size_thdr_size;structsocket*sock;+intbytes_coalesced=0;+intbufs_coalesced=0;+intpackets_coalesced=0;/* TODO: check that we are running from vhost_worker? */sock=rcu_dereference_check(vq->private_data,1);
From: Steve Dobbelstein <hidden> Date: 2011-01-28 18:29:44
mashirle@linux.vnet.ibm.com wrote on 01/27/2011 02:15:05 PM:
On Thu, 2011-01-27 at 22:05 +0200, Michael S. Tsirkin wrote:
quoted
One simple theory is that guest net stack became faster
and so the host can't keep up.
Yes, that's what I think here. Some qdisc code has been changed
recently.
I ran a test with txqueuelen set to 128, instead of the default of 1000, in
the guest in an attempt to slow down the guest transmits. The change had
no effect on the throughput nor on the CPU usage.
On the other hand, I ran some tests with different CPU pinnings and
with/without hyperthreading enabled. Here is a summary of the results.
Pinning configuration 1: pin the VCPUs and pin the vhost thread to one of
the VCPU CPUs
Pinning configuration 2: pin the VCPUs and pin the vhost thread to a
separate CPU on the same socket
Pinning configuration 3: pin the VCPUs and pin the vhost thread to a
separate CPU a different socket
HT Pinning Throughput CPU
Yes config 1 - 40% - 40%
Yes config 2 - 37% - 35%
Yes config 3 - 37% - 36%
No none 0% - 5%
No config 1 - 41% - 43%
No config 2 + 32% - 4%
No config 3 + 34% + 9%
Pinning the vhost thread to the same CPU as a guest VCPU hurts performance.
Turning off hyperthreading and pinning the VPUS and vhost thread to
separate CPUs significantly improves performance, getting it into the
competitive range with other hypervisors.
Steve D.
From: Steve Dobbelstein <hidden> Date: 2011-01-28 22:52:06
steved@us.ibm.com wrote on 01/28/2011 12:29:37 PM:
quoted
On Thu, 2011-01-27 at 22:05 +0200, Michael S. Tsirkin wrote:
quoted
One simple theory is that guest net stack became faster
and so the host can't keep up.
Yes, that's what I think here. Some qdisc code has been changed
recently.
I ran a test with txqueuelen set to 128, instead of the default of 1000,
in
the guest in an attempt to slow down the guest transmits. The change had
no effect on the throughput nor on the CPU usage.
On the other hand, I ran some tests with different CPU pinnings and
with/without hyperthreading enabled. Here is a summary of the results.
Pinning configuration 1: pin the VCPUs and pin the vhost thread to one
of
the VCPU CPUs
Pinning configuration 2: pin the VCPUs and pin the vhost thread to a
separate CPU on the same socket
Pinning configuration 3: pin the VCPUs and pin the vhost thread to a
separate CPU a different socket
HT Pinning Throughput CPU
Yes config 1 - 40% - 40%
Yes config 2 - 37% - 35%
Yes config 3 - 37% - 36%
No none 0% - 5%
No config 1 - 41% - 43%
No config 2 + 32% - 4%
No config 3 + 34% + 9%
Pinning the vhost thread to the same CPU as a guest VCPU hurts
performance.
Turning off hyperthreading and pinning the VPUS and vhost thread to
separate CPUs significantly improves performance, getting it into the
competitive range with other hypervisors.
Steve D.
Those results for configs 2 and 3 with hyperthreading off are a little
strange. Digging into the cause I found that my automation script for
pinning the vhost thread failed and pinned it to CPU 1, the same as config
1, giving results similar to config 1. I reran the tests making sure the
pinning script did the right thing. The results are more consistent.
HT Pinning Throughput CPU
Yes config 1 - 40% - 40%
Yes config 2 + 33% - 8%
Yes config 3 + 34% + 9%
No none 0% - 5%
No config 1 - 41% - 43%
No config 2 + 32% - 4%
No config 3 + 34% + 9%
It appears that we have a scheduling problem. If the processes are pinned
we can get good performance.
We also se that hyperthreading makes little difference.
Sorry for the initial misleading data.
Steve D.
From: Steve Dobbelstein <hidden> Date: 2011-02-01 00:24:38
"Michael S. Tsirkin" [off-list ref] wrote on 01/28/2011 06:16:16 AM:
quoted hunk
OK, so thinking about it more, maybe the issue is this:
tx becomes full. We process one request and interrupt the guest,
then it adds one request and the queue is full again.
Maybe the following will help it stabilize?
By itself it does nothing, but if you set
all the parameters to a huge value we will
only interrupt when we see an empty ring.
Which might be too much: pls try other values
in the middle: e.g. make bufs half the ring,
or bytes some small value, or packets some
small value etc.
Warning: completely untested.
@@ -127,6 +134,9 @@ static void handle_tx(struct vhost_net *net)interr,wmem;size_thdr_size;structsocket*sock;+intbytes_coalesced=0;+intbufs_coalesced=0;+intpackets_coalesced=0;/* TODO: check that we are running from vhost_worker? */sock=rcu_dereference_check(vq->private_data,1);
Should this instead be:
bufs_coalesced += out;
Perusing the code I see that earlier there is a check to see if "in" is not
zero, and, if so, error out of the loop. After the check, "in" is not
touched until it is added to bufs_coalesced, effectively not changing
bufs_coalesced, meaning bufs_coalesced will never trigger the conditions
below.
Or am I missing something?
On Mon, 2011-01-31 at 18:24 -0600, Steve Dobbelstein wrote:
"Michael S. Tsirkin" [off-list ref] wrote on 01/28/2011 06:16:16 AM:
quoted
OK, so thinking about it more, maybe the issue is this:
tx becomes full. We process one request and interrupt the guest,
then it adds one request and the queue is full again.
Maybe the following will help it stabilize?
By itself it does nothing, but if you set
all the parameters to a huge value we will
only interrupt when we see an empty ring.
Which might be too much: pls try other values
in the middle: e.g. make bufs half the ring,
or bytes some small value, or packets some
small value etc.
Warning: completely untested.
@@ -127,6 +134,9 @@ static void handle_tx(struct vhost_net *net)interr,wmem;size_thdr_size;structsocket*sock;+intbytes_coalesced=0;+intbufs_coalesced=0;+intpackets_coalesced=0;/* TODO: check that we are running from vhost_worker? */sock=rcu_dereference_check(vq->private_data,1);
Should this instead be:
bufs_coalesced += out;
Perusing the code I see that earlier there is a check to see if "in" is not
zero, and, if so, error out of the loop. After the check, "in" is not
touched until it is added to bufs_coalesced, effectively not changing
bufs_coalesced, meaning bufs_coalesced will never trigger the conditions
below.
Yes. It definitely should be 'out'. 'in' should be 0 in the tx path.
I tried a simpler version of this patch without any tunables by
delaying the signaling until we come out of the for loop.
It definitely reduced the number of vmexits significantly for small message
guest to host stream test and the throughput went up a little.
It is possible that we can miss signaling the guest even after
processing a few pkts, if we don't hit any of these conditions.
quoted
Steve D.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2011-02-01 05:54:45
On Mon, Jan 31, 2011 at 06:24:34PM -0600, Steve Dobbelstein wrote:
"Michael S. Tsirkin" [off-list ref] wrote on 01/28/2011 06:16:16 AM:
quoted
OK, so thinking about it more, maybe the issue is this:
tx becomes full. We process one request and interrupt the guest,
then it adds one request and the queue is full again.
Maybe the following will help it stabilize?
By itself it does nothing, but if you set
all the parameters to a huge value we will
only interrupt when we see an empty ring.
Which might be too much: pls try other values
in the middle: e.g. make bufs half the ring,
or bytes some small value, or packets some
small value etc.
Warning: completely untested.
@@ -127,6 +134,9 @@ static void handle_tx(struct vhost_net *net)interr,wmem;size_thdr_size;structsocket*sock;+intbytes_coalesced=0;+intbufs_coalesced=0;+intpackets_coalesced=0;/* TODO: check that we are running from vhost_worker? */sock=rcu_dereference_check(vq->private_data,1);
Perusing the code I see that earlier there is a check to see if "in" is not
zero, and, if so, error out of the loop. After the check, "in" is not
touched until it is added to bufs_coalesced, effectively not changing
bufs_coalesced, meaning bufs_coalesced will never trigger the conditions
below.
Or am I missing something?
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2011-02-01 05:56:27
On Mon, Jan 31, 2011 at 05:30:38PM -0800, Sridhar Samudrala wrote:
quoted hunk
On Mon, 2011-01-31 at 18:24 -0600, Steve Dobbelstein wrote:
quoted
"Michael S. Tsirkin" [off-list ref] wrote on 01/28/2011 06:16:16 AM:
quoted
OK, so thinking about it more, maybe the issue is this:
tx becomes full. We process one request and interrupt the guest,
then it adds one request and the queue is full again.
Maybe the following will help it stabilize?
By itself it does nothing, but if you set
all the parameters to a huge value we will
only interrupt when we see an empty ring.
Which might be too much: pls try other values
in the middle: e.g. make bufs half the ring,
or bytes some small value, or packets some
small value etc.
Warning: completely untested.
@@ -127,6 +134,9 @@ static void handle_tx(struct vhost_net *net)interr,wmem;size_thdr_size;structsocket*sock;+intbytes_coalesced=0;+intbufs_coalesced=0;+intpackets_coalesced=0;/* TODO: check that we are running from vhost_worker? */sock=rcu_dereference_check(vq->private_data,1);
Should this instead be:
bufs_coalesced += out;
Perusing the code I see that earlier there is a check to see if "in" is not
zero, and, if so, error out of the loop. After the check, "in" is not
touched until it is added to bufs_coalesced, effectively not changing
bufs_coalesced, meaning bufs_coalesced will never trigger the conditions
below.
Yes. It definitely should be 'out'. 'in' should be 0 in the tx path.
I tried a simpler version of this patch without any tunables by
delaying the signaling until we come out of the for loop.
It definitely reduced the number of vmexits significantly for small message
guest to host stream test and the throughput went up a little.
It is possible that we can miss signaling the guest even after
processing a few pkts, if we don't hit any of these conditions.
Yes. It really should be
if (likely(packets_coalesced && bytes_coalesced && bufs_coalesced))
vhost_signal(&net->dev, vq);
quoted
quoted
Steve D.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2011-02-01 15:52:53
OK, so thinking about it more, maybe the issue is this:
tx becomes full. We process one request and interrupt the guest,
then it adds one request and the queue is full again.
Maybe the following will help it stabilize? By default with it we will
only interrupt when we see an empty ring.
Which is liklely too much: pls try other values
in the middle: e.g. make bufs half the ring,
or bytes some small value like half ring * 200, or packets some
small value etc.
Set any one parameter to 0 to get current
behaviour (interrupt immediately when enabled).
Warning: completely untested.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
@@ -127,6 +134,9 @@ static void handle_tx(struct vhost_net *net)interr,wmem;size_thdr_size;structsocket*sock;+intbytes_coalesced=0;+intbufs_coalesced=0;+intpackets_coalesced=0;/* TODO: check that we are running from vhost_worker? */sock=rcu_dereference_check(vq->private_data,1);
From: Shirley Ma <hidden> Date: 2011-02-01 21:09:45
On Mon, 2011-01-31 at 17:30 -0800, Sridhar Samudrala wrote:
quoted hunk
Yes. It definitely should be 'out'. 'in' should be 0 in the tx path.
I tried a simpler version of this patch without any tunables by
delaying the signaling until we come out of the for loop.
It definitely reduced the number of vmexits significantly for small
message
guest to host stream test and the throughput went up a little.
Reducing the signaling will reduce the CPU utilization by reducing VM
exits.
The small message BW is a problem we have seen faster guest/slow vhost,
even I increased VHOST_NET_WEIGHT times, it didn't help that much for
BW. For large message size, vhost is able to process all packets on
time. I played around with guest/host codes, I only see huge BW
improvement by dropping packets on guest side so far.
Thanks
Shirley
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2011-02-01 21:24:11
On Tue, Feb 01, 2011 at 01:09:45PM -0800, Shirley Ma wrote:
On Mon, 2011-01-31 at 17:30 -0800, Sridhar Samudrala wrote:
quoted
Yes. It definitely should be 'out'. 'in' should be 0 in the tx path.
I tried a simpler version of this patch without any tunables by
delaying the signaling until we come out of the for loop.
It definitely reduced the number of vmexits significantly for small
message
guest to host stream test and the throughput went up a little.
Reducing the signaling will reduce the CPU utilization by reducing VM
exits.
The small message BW is a problem we have seen faster guest/slow vhost,
even I increased VHOST_NET_WEIGHT times, it didn't help that much for
BW. For large message size, vhost is able to process all packets on
time. I played around with guest/host codes, I only see huge BW
improvement by dropping packets on guest side so far.
Thanks
Shirley
My theory is that the issue is not signalling.
Rather, our queue fills up, then host handles
one packet and sends an interrupt, and we
immediately wake the queue. So the vq
once it gets full, stays full.
If you try my patch with bufs threshold set to e.g.
half the vq, what we will do is send interrupt after we have processed
half the vq. So host has half the vq to go, and guest has half the vq
to fill.
See?
--
MST
From: Shirley Ma <hidden> Date: 2011-02-01 21:32:35
On Tue, 2011-02-01 at 23:24 +0200, Michael S. Tsirkin wrote:
My theory is that the issue is not signalling.
Rather, our queue fills up, then host handles
one packet and sends an interrupt, and we
immediately wake the queue. So the vq
once it gets full, stays full.
From the printk debugging output, it might not be exactly the case. The
ring gets full, run a bit, then gets full, then run a bit, then full...
If you try my patch with bufs threshold set to e.g.
half the vq, what we will do is send interrupt after we have processed
half the vq. So host has half the vq to go, and guest has half the vq
to fill.
See?
I am cleaning up my set up to run your patch ...
Shirley
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2011-02-01 21:42:28
On Tue, Feb 01, 2011 at 01:32:35PM -0800, Shirley Ma wrote:
On Tue, 2011-02-01 at 23:24 +0200, Michael S. Tsirkin wrote:
quoted
My theory is that the issue is not signalling.
Rather, our queue fills up, then host handles
one packet and sends an interrupt, and we
immediately wake the queue. So the vq
once it gets full, stays full.
quoted
From the printk debugging output, it might not be exactly the case. The
ring gets full, run a bit, then gets full, then run a bit, then full...
Yes, but does it get even half empty in between?
quoted
If you try my patch with bufs threshold set to e.g.
half the vq, what we will do is send interrupt after we have processed
half the vq. So host has half the vq to go, and guest has half the vq
to fill.
See?
I am cleaning up my set up to run your patch ...
Shirley
From: Shirley Ma <hidden> Date: 2011-02-01 21:53:09
On Tue, 2011-02-01 at 23:42 +0200, Michael S. Tsirkin wrote:
On Tue, Feb 01, 2011 at 01:32:35PM -0800, Shirley Ma wrote:
quoted
On Tue, 2011-02-01 at 23:24 +0200, Michael S. Tsirkin wrote:
quoted
My theory is that the issue is not signalling.
Rather, our queue fills up, then host handles
one packet and sends an interrupt, and we
immediately wake the queue. So the vq
once it gets full, stays full.
quoted
From the printk debugging output, it might not be exactly the case.
The
quoted
ring gets full, run a bit, then gets full, then run a bit, then
full...
Yes, but does it get even half empty in between?
Sometimes, most of them not half of empty in between. But printk slow
down the traffics, so it's not accurate. I think your patch will improve
the performance if it signals guest when half of the ring size is
empty.
But you manage signal by using TX bytes, I would like to change it to
half of the ring size instead for signaling. Is that OK?
Shirley
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2011-02-01 21:56:03
On Tue, Feb 01, 2011 at 01:53:05PM -0800, Shirley Ma wrote:
On Tue, 2011-02-01 at 23:42 +0200, Michael S. Tsirkin wrote:
quoted
On Tue, Feb 01, 2011 at 01:32:35PM -0800, Shirley Ma wrote:
quoted
On Tue, 2011-02-01 at 23:24 +0200, Michael S. Tsirkin wrote:
quoted
My theory is that the issue is not signalling.
Rather, our queue fills up, then host handles
one packet and sends an interrupt, and we
immediately wake the queue. So the vq
once it gets full, stays full.
quoted
From the printk debugging output, it might not be exactly the case.
The
quoted
ring gets full, run a bit, then gets full, then run a bit, then
full...
Yes, but does it get even half empty in between?
Sometimes, most of them not half of empty in between. But printk slow
down the traffics, so it's not accurate. I think your patch will improve
the performance if it signals guest when half of the ring size is
empty.
But you manage signal by using TX bytes,
There are flags for bytes, buffers and packets.
Try playing with any one of them :)
Just be sure to use v2.
I would like to change it to
half of the ring size instead for signaling. Is that OK?
Shirley
Sure that is why I made it a parameter so you can experiment.
--
MST
From: Shirley Ma <hidden> Date: 2011-02-01 22:59:57
On Tue, 2011-02-01 at 23:56 +0200, Michael S. Tsirkin wrote:
There are flags for bytes, buffers and packets.
Try playing with any one of them :)
Just be sure to use v2.
quoted
I would like to change it to
half of the ring size instead for signaling. Is that OK?
Shirley
Sure that is why I made it a parameter so you can experiment.
The initial test results shows that the CPUs utilization has been
reduced some, and BW has increased some with the default parameters,
like 1K message size BW goes from 2.5Gb/s about 2.8Gb/s, CPU utilization
down from 4x% to 38%, (Similar results from the patch I submitted a
while ago to reduce signaling on vhost) but far away from dropping
packet results.
I am going to change the code to use 1/2 ring size to wake the netif
queue.
Shirley
On Tue, 2011-02-01 at 17:52 +0200, Michael S. Tsirkin wrote:
quoted hunk
OK, so thinking about it more, maybe the issue is this:
tx becomes full. We process one request and interrupt the guest,
then it adds one request and the queue is full again.
Maybe the following will help it stabilize? By default with it we will
only interrupt when we see an empty ring.
Which is liklely too much: pls try other values
in the middle: e.g. make bufs half the ring,
or bytes some small value like half ring * 200, or packets some
small value etc.
Set any one parameter to 0 to get current
behaviour (interrupt immediately when enabled).
Warning: completely untested.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
@@ -127,6 +134,9 @@ static void handle_tx(struct vhost_net *net)interr,wmem;size_thdr_size;structsocket*sock;+intbytes_coalesced=0;+intbufs_coalesced=0;+intpackets_coalesced=0;/* TODO: check that we are running from vhost_worker? */sock=rcu_dereference_check(vq->private_data,1);
I think the counters that exceed the limits need to be reset to 0 here.
Otherwise we keep signaling for every buffer once we hit this condition.
Thanks
Sridhar
+ else
+ vhost_add_used(vq, head, 0);
if (unlikely(total_len >= VHOST_NET_WEIGHT)) {
vhost_poll_queue(&vq->poll);
break;
}
}
+ if (likely(packets_coalesced &&
+ bytes_coalesced &&
+ bufs_coalesced))
+ vhost_signal(&net->dev, vq);
mutex_unlock(&vq->mutex);
}
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2011-02-02 04:36:28
On Tue, Feb 01, 2011 at 03:07:38PM -0800, Sridhar Samudrala wrote:
On Tue, 2011-02-01 at 17:52 +0200, Michael S. Tsirkin wrote:
quoted
OK, so thinking about it more, maybe the issue is this:
tx becomes full. We process one request and interrupt the guest,
then it adds one request and the queue is full again.
Maybe the following will help it stabilize? By default with it we will
only interrupt when we see an empty ring.
Which is liklely too much: pls try other values
in the middle: e.g. make bufs half the ring,
or bytes some small value like half ring * 200, or packets some
small value etc.
Set any one parameter to 0 to get current
behaviour (interrupt immediately when enabled).
Warning: completely untested.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
@@ -127,6 +134,9 @@ static void handle_tx(struct vhost_net *net)interr,wmem;size_thdr_size;structsocket*sock;+intbytes_coalesced=0;+intbufs_coalesced=0;+intpackets_coalesced=0;/* TODO: check that we are running from vhost_worker? */sock=rcu_dereference_check(vq->private_data,1);
I think the counters that exceed the limits need to be reset to 0 here.
Otherwise we keep signaling for every buffer once we hit this condition.
Thanks
Sridhar
Correct, good catch.
quoted
+ else
+ vhost_add_used(vq, head, 0);
if (unlikely(total_len >= VHOST_NET_WEIGHT)) {
vhost_poll_queue(&vq->poll);
break;
}
}
+ if (likely(packets_coalesced &&
+ bytes_coalesced &&
+ bufs_coalesced))
+ vhost_signal(&net->dev, vq);
mutex_unlock(&vq->mutex);
}
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2011-02-02 04:40:21
On Tue, Feb 01, 2011 at 02:59:57PM -0800, Shirley Ma wrote:
On Tue, 2011-02-01 at 23:56 +0200, Michael S. Tsirkin wrote:
quoted
There are flags for bytes, buffers and packets.
Try playing with any one of them :)
Just be sure to use v2.
quoted
I would like to change it to
half of the ring size instead for signaling. Is that OK?
Shirley
Sure that is why I made it a parameter so you can experiment.
The initial test results shows that the CPUs utilization has been
reduced some, and BW has increased some with the default parameters,
like 1K message size BW goes from 2.5Gb/s about 2.8Gb/s, CPU utilization
down from 4x% to 38%, (Similar results from the patch I submitted a
while ago to reduce signaling on vhost) but far away from dropping
packet results.
I am going to change the code to use 1/2 ring size to wake the netif
queue.
Shirley
Just tweak the parameters with sysfs, you do not have to edit the code:
echo 64 > /sys/module/vhost_net/parameters/tx_bufs_coalesce
Or in a similar way for tx_packets_coalesce (since we use indirect,
packets will typically use 1 buffer each).
--
MST
From: Shirley Ma <hidden> Date: 2011-02-02 06:06:03
On Wed, 2011-02-02 at 06:40 +0200, Michael S. Tsirkin wrote:
ust tweak the parameters with sysfs, you do not have to edit the code:
echo 64 > /sys/module/vhost_net/parameters/tx_bufs_coalesce
Or in a similar way for tx_packets_coalesce (since we use indirect,
packets will typically use 1 buffer each).
We should use packets instead of buffers, in indirect case, one packet
has multiple buffers, each packet uses one descriptor from the ring
(default size is 256).
echo 128 > /sys/module/vhost_net/parameters/tx_packets_coalesce
The way I am changing is only when netif queue has stopped, then we
start to count num_free descriptors to send the signal to wake netif
queue.
Shirley
From: Shirley Ma <hidden> Date: 2011-02-02 06:19:09
On Tue, 2011-02-01 at 22:05 -0800, Shirley Ma wrote:
The way I am changing is only when netif queue has stopped, then we
start to count num_free descriptors to send the signal to wake netif
queue.
I forgot to mention, the code change I am making is in guest kernel, in
xmit call back only wake up the queue when it's stopped && num_free >=
1/2 *vq->num, I add a new API in virtio_ring.
However vhost signaling reduction is needed as well. The patch I
submitted a while ago showed both CPUs and BW improvement.
Thanks
Shirley
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2011-02-02 06:30:07
On Tue, Feb 01, 2011 at 10:19:09PM -0800, Shirley Ma wrote:
On Tue, 2011-02-01 at 22:05 -0800, Shirley Ma wrote:
quoted
The way I am changing is only when netif queue has stopped, then we
start to count num_free descriptors to send the signal to wake netif
queue.
I forgot to mention, the code change I am making is in guest kernel, in
xmit call back only wake up the queue when it's stopped && num_free >=
1/2 *vq->num, I add a new API in virtio_ring.
Interesting. Yes, I agree an API extension would be helpful. However,
wouldn't just the signaling reduction be enough, without guest changes?
However vhost signaling reduction is needed as well. The patch I
submitted a while ago showed both CPUs and BW improvement.
Thanks
Shirley
On Tue, 2011-02-01 at 22:05 -0800, Shirley Ma wrote:
quoted
The way I am changing is only when netif queue has stopped, then we
start to count num_free descriptors to send the signal to wake netif
queue.
I forgot to mention, the code change I am making is in guest kernel, in
xmit call back only wake up the queue when it's stopped && num_free >=
1/2 *vq->num, I add a new API in virtio_ring.
FYI :)
I have tried this before. There are a couple of issues:
1. the free count will not reduce until you run free_old_xmit_skbs,
which will not run anymore since the tx queue is stopped.
2. You cannot call free_old_xmit_skbs directly as it races with a
queue that was just awakened (current cb was due to the delay
in disabling cb's).
You have to call free_old_xmit_skbs() under netif_queue_stopped()
check to avoid the race.
I got a small improvement in my testing upto some number of threads
(32 or 48?), but beyond that I was getting a regression.
Thanks,
- KK
However vhost signaling reduction is needed as well. The patch I
submitted a while ago showed both CPUs and BW improvement.
From: Shirley Ma <hidden> Date: 2011-02-02 07:03:51
On Wed, 2011-02-02 at 12:04 +0530, Krishna Kumar2 wrote:
quoted
On Tue, 2011-02-01 at 22:05 -0800, Shirley Ma wrote:
quoted
The way I am changing is only when netif queue has stopped, then
we
quoted
quoted
start to count num_free descriptors to send the signal to wake
netif
quoted
quoted
queue.
I forgot to mention, the code change I am making is in guest kernel,
in
quoted
xmit call back only wake up the queue when it's stopped && num_free
=
1/2 *vq->num, I add a new API in virtio_ring.
FYI :)
I have tried this before. There are a couple of issues:
1. the free count will not reduce until you run free_old_xmit_skbs,
which will not run anymore since the tx queue is stopped.
2. You cannot call free_old_xmit_skbs directly as it races with a
queue that was just awakened (current cb was due to the delay
in disabling cb's).
You have to call free_old_xmit_skbs() under netif_queue_stopped()
check to avoid the race.
Yes, that' what I did, when the netif queue stop, don't enable the
queue, just free_old_xmit_skbs(), if not enough freed, then enabling
callback until half of the ring size are freed, then wake the netif
queue. But somehow I didn't reach the performance compared to drop
packets, need to think about it more. :)
Thanks
Shirley
From: Shirley Ma <hidden> Date: 2011-02-02 07:14:51
On Wed, 2011-02-02 at 08:29 +0200, Michael S. Tsirkin wrote:
On Tue, Feb 01, 2011 at 10:19:09PM -0800, Shirley Ma wrote:
quoted
On Tue, 2011-02-01 at 22:05 -0800, Shirley Ma wrote:
quoted
The way I am changing is only when netif queue has stopped, then
we
quoted
quoted
start to count num_free descriptors to send the signal to wake
netif
quoted
quoted
queue.
I forgot to mention, the code change I am making is in guest kernel,
in
quoted
xmit call back only wake up the queue when it's stopped && num_free
=
1/2 *vq->num, I add a new API in virtio_ring.
Interesting. Yes, I agree an API extension would be helpful. However,
wouldn't just the signaling reduction be enough, without guest
changes?
w/i guest change, I played around the parameters,for example: I could
get 3.7Gb/s with 42% CPU BW increasing from 2.5Gb/s for 1K message size,
w/i dropping packet, I was able to get up to 6.2Gb/s with similar CPU
usage.
quoted
However vhost signaling reduction is needed as well. The patch I
submitted a while ago showed both CPUs and BW improvement.
Thanks
Shirley
Which patch was that?
The patch was called "vhost: TX used buffer guest signal accumulation".
You suggested to split add_used_bufs and signal. I am still thinking
what's the best approach to cooperate guest (virtio_kick) and
vhost(handle_tx), vhost(signaling) and guest (xmit callback) to reduce
the overheads, so I haven't submit the new patch yet.
Thanks
Shirley
From: Shirley Ma <hidden> Date: 2011-02-02 07:33:57
On Tue, 2011-02-01 at 23:14 -0800, Shirley Ma wrote:
w/i guest change, I played around the parameters,for example: I could
get 3.7Gb/s with 42% CPU BW increasing from 2.5Gb/s for 1K message
size,
w/i dropping packet, I was able to get up to 6.2Gb/s with similar CPU
usage.
I meant w/o guest change, only vhost changes. Sorry about that.
Shirley
I have tried this before. There are a couple of issues:
1. the free count will not reduce until you run free_old_xmit_skbs,
which will not run anymore since the tx queue is stopped.
2. You cannot call free_old_xmit_skbs directly as it races with a
queue that was just awakened (current cb was due to the delay
in disabling cb's).
You have to call free_old_xmit_skbs() under netif_queue_stopped()
check to avoid the race.
Yes, that' what I did, when the netif queue stop, don't enable the
queue, just free_old_xmit_skbs(), if not enough freed, then enabling
callback until half of the ring size are freed, then wake the netif
queue. But somehow I didn't reach the performance compared to drop
packets, need to think about it more. :)
Did you check if the number of vmexits increased with this
patch? This is possible if the device was keeping up (and
not going into a stop, start, xmit 1 packet, stop, start
loop). Also maybe you should try for 1/4th instead of 1/2?
MST's delayed signalling should avoid this issue, I haven't
tried both together.
Thanks,
- KK
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2011-02-02 10:48:33
On Wed, Feb 02, 2011 at 12:04:37PM +0530, Krishna Kumar2 wrote:
quoted
On Tue, 2011-02-01 at 22:05 -0800, Shirley Ma wrote:
quoted
The way I am changing is only when netif queue has stopped, then we
start to count num_free descriptors to send the signal to wake netif
queue.
I forgot to mention, the code change I am making is in guest kernel, in
xmit call back only wake up the queue when it's stopped && num_free >=
1/2 *vq->num, I add a new API in virtio_ring.
FYI :)
I have tried this before. There are a couple of issues:
1. the free count will not reduce until you run free_old_xmit_skbs,
which will not run anymore since the tx queue is stopped.
2. You cannot call free_old_xmit_skbs directly as it races with a
queue that was just awakened (current cb was due to the delay
in disabling cb's).
You have to call free_old_xmit_skbs() under netif_queue_stopped()
check to avoid the race.
I got a small improvement in my testing upto some number of threads
(32 or 48?), but beyond that I was getting a regression.
Thanks,
- KK
quoted
However vhost signaling reduction is needed as well. The patch I
submitted a while ago showed both CPUs and BW improvement.
Yes, I think doing this in the host is much simpler,
just send an interrupt after there's a decent amount
of space in the queue.
Having said that the simple heuristic that I coded
might be a bit too simple.
--
MST
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2011-02-02 10:48:43
On Tue, Feb 01, 2011 at 11:14:51PM -0800, Shirley Ma wrote:
On Wed, 2011-02-02 at 08:29 +0200, Michael S. Tsirkin wrote:
quoted
On Tue, Feb 01, 2011 at 10:19:09PM -0800, Shirley Ma wrote:
quoted
On Tue, 2011-02-01 at 22:05 -0800, Shirley Ma wrote:
quoted
The way I am changing is only when netif queue has stopped, then
we
quoted
quoted
start to count num_free descriptors to send the signal to wake
netif
quoted
quoted
queue.
I forgot to mention, the code change I am making is in guest kernel,
in
quoted
xmit call back only wake up the queue when it's stopped && num_free
=
1/2 *vq->num, I add a new API in virtio_ring.
Interesting. Yes, I agree an API extension would be helpful. However,
wouldn't just the signaling reduction be enough, without guest
changes?
w/i guest change, I played around the parameters,for example: I could
get 3.7Gb/s with 42% CPU BW increasing from 2.5Gb/s for 1K message size,
w/i dropping packet, I was able to get up to 6.2Gb/s with similar CPU
usage.
We need to consider them separately IMO. What's the best we can get
without guest change? And which parameters give it?
There will always be old guests, and as far as I can tell
it should work better from host.
quoted
quoted
However vhost signaling reduction is needed as well. The patch I
submitted a while ago showed both CPUs and BW improvement.
Thanks
Shirley
Which patch was that?
The patch was called "vhost: TX used buffer guest signal accumulation".
Yes, a somewhat similar idea.
You suggested to split add_used_bufs and signal.
Exactly. And this is basically what this patch does.
I am still thinking
what's the best approach to cooperate guest (virtio_kick) and
vhost(handle_tx), vhost(signaling) and guest (xmit callback) to reduce
the overheads, so I haven't submit the new patch yet.
Thanks
Shirley
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2011-02-02 10:50:05
On Tue, Feb 01, 2011 at 11:33:49PM -0800, Shirley Ma wrote:
On Tue, 2011-02-01 at 23:14 -0800, Shirley Ma wrote:
quoted
w/i guest change, I played around the parameters,for example: I could
get 3.7Gb/s with 42% CPU BW increasing from 2.5Gb/s for 1K message
size,
w/i dropping packet, I was able to get up to 6.2Gb/s with similar CPU
usage.
I meant w/o guest change, only vhost changes. Sorry about that.
Shirley
From: Shirley Ma <hidden> Date: 2011-02-02 15:39:45
On Wed, 2011-02-02 at 12:48 +0200, Michael S. Tsirkin wrote:
Yes, I think doing this in the host is much simpler,
just send an interrupt after there's a decent amount
of space in the queue.
Having said that the simple heuristic that I coded
might be a bit too simple.
From the debugging out what I have seen so far (a single small message
TCP_STEAM test), I think the right approach is to patch both guest and
vhost. The problem I have found is a regression for single small
message TCP_STEAM test. Old kernel works well for TCP_STREAM, only new
kernel has problem.
For Steven's problem, it's multiple stream TCP_RR issues, the old guest
doesn't perform well, so does new guest kernel. We tested reducing vhost
signaling patch before, it didn't help the performance at all.
Thanks
Shirley
From: Shirley Ma <hidden> Date: 2011-02-02 15:44:15
On Wed, 2011-02-02 at 12:49 +0200, Michael S. Tsirkin wrote:
On Tue, Feb 01, 2011 at 11:33:49PM -0800, Shirley Ma wrote:
quoted
On Tue, 2011-02-01 at 23:14 -0800, Shirley Ma wrote:
quoted
w/i guest change, I played around the parameters,for example: I
could
quoted
quoted
get 3.7Gb/s with 42% CPU BW increasing from 2.5Gb/s for 1K message
size,
w/i dropping packet, I was able to get up to 6.2Gb/s with similar
CPU
quoted
quoted
usage.
I meant w/o guest change, only vhost changes. Sorry about that.
Shirley
Ah, excellent. What were the parameters?
I used half of the ring size 129 for packet counters, but the
performance is still not as good as dropping packets on guest, 3.7 Gb/s
vs. 6.2Gb/s.
Shirley
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2011-02-02 15:47:26
On Wed, Feb 02, 2011 at 07:39:45AM -0800, Shirley Ma wrote:
On Wed, 2011-02-02 at 12:48 +0200, Michael S. Tsirkin wrote:
quoted
Yes, I think doing this in the host is much simpler,
just send an interrupt after there's a decent amount
of space in the queue.
Having said that the simple heuristic that I coded
might be a bit too simple.
quoted
From the debugging out what I have seen so far (a single small message
TCP_STEAM test), I think the right approach is to patch both guest and
vhost.
One problem is slowing down the guest helps here.
So there's a chance that just by adding complexity
in guest driver we get a small improvement :(
We can't rely on a patched guest anyway, so
I think it is best to test guest and host changes separately.
And I do agree something needs to be done in guest too,
for example when vqs share an interrupt, we
might invoke a callback when we see vq is not empty
even though it's not requested. Probably should
check interrupts enabled here?
The problem I have found is a regression for single small
message TCP_STEAM test. Old kernel works well for TCP_STREAM, only new
kernel has problem.
Likely new kernel is faster :)
For Steven's problem, it's multiple stream TCP_RR issues, the old guest
doesn't perform well, so does new guest kernel. We tested reducing vhost
signaling patch before, it didn't help the performance at all.
Thanks
Shirley
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2011-02-02 15:48:41
On Wed, Feb 02, 2011 at 07:42:51AM -0800, Shirley Ma wrote:
On Wed, 2011-02-02 at 12:49 +0200, Michael S. Tsirkin wrote:
quoted
On Tue, Feb 01, 2011 at 11:33:49PM -0800, Shirley Ma wrote:
quoted
On Tue, 2011-02-01 at 23:14 -0800, Shirley Ma wrote:
quoted
w/i guest change, I played around the parameters,for example: I
could
quoted
quoted
get 3.7Gb/s with 42% CPU BW increasing from 2.5Gb/s for 1K message
size,
w/i dropping packet, I was able to get up to 6.2Gb/s with similar
CPU
quoted
quoted
usage.
I meant w/o guest change, only vhost changes. Sorry about that.
Shirley
Ah, excellent. What were the parameters?
I used half of the ring size 129 for packet counters, but the
performance is still not as good as dropping packets on guest, 3.7 Gb/s
vs. 6.2Gb/s.
Shirley
And this is with sndbuf=0 in host, yes?
And do you see a lot of tx interrupts?
How packets per interrupt?
--
MST
From: Shirley Ma <hidden> Date: 2011-02-02 17:10:35
On Wed, 2011-02-02 at 17:47 +0200, Michael S. Tsirkin wrote:
On Wed, Feb 02, 2011 at 07:39:45AM -0800, Shirley Ma wrote:
quoted
On Wed, 2011-02-02 at 12:48 +0200, Michael S. Tsirkin wrote:
quoted
Yes, I think doing this in the host is much simpler,
just send an interrupt after there's a decent amount
of space in the queue.
Having said that the simple heuristic that I coded
might be a bit too simple.
quoted
From the debugging out what I have seen so far (a single small
message
quoted
TCP_STEAM test), I think the right approach is to patch both guest
and
quoted
vhost.
One problem is slowing down the guest helps here.
So there's a chance that just by adding complexity
in guest driver we get a small improvement :(
We can't rely on a patched guest anyway, so
I think it is best to test guest and host changes separately.
And I do agree something needs to be done in guest too,
for example when vqs share an interrupt, we
might invoke a callback when we see vq is not empty
even though it's not requested. Probably should
check interrupts enabled here?
Yes, I modified xmit callback something like below:
static void skb_xmit_done(struct virtqueue *svq)
{
struct virtnet_info *vi = svq->vdev->priv;
/* Suppress further interrupts. */
virtqueue_disable_cb(svq);
/* We were probably waiting for more output buffers. */
if (netif_queue_stopped(vi->dev)) {
free_old_xmit_skbs(vi);
if (virtqueue_free_size(svq) <= svq->vring.num / 2) {
virtqueue_enable_cb(svq);
return;
}
}
netif_wake_queue(vi->dev);
}
quoted
The problem I have found is a regression for single small
message TCP_STEAM test. Old kernel works well for TCP_STREAM, only
new
quoted
kernel has problem.
Likely new kernel is faster :)
quoted
For Steven's problem, it's multiple stream TCP_RR issues, the old
guest
quoted
doesn't perform well, so does new guest kernel. We tested reducing
vhost
quoted
signaling patch before, it didn't help the performance at all.
Thanks
Shirley
Yes, it seems unrelated to tx interrupts.
The issue is more likely related to latency. Do you have anything in
mind on how to reduce vhost latency?
Thanks
Shirley
From: Shirley Ma <hidden> Date: 2011-02-02 17:13:02
On Wed, 2011-02-02 at 17:48 +0200, Michael S. Tsirkin wrote:
And this is with sndbuf=0 in host, yes?
And do you see a lot of tx interrupts?
How packets per interrupt?
Nope, sndbuf doens't matter since I never hit reaching sock wmem
condition in vhost. I am still playing around, let me know what data you
would like to collect.
Thanks
Shirley
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2011-02-02 17:32:42
On Wed, Feb 02, 2011 at 09:10:35AM -0800, Shirley Ma wrote:
On Wed, 2011-02-02 at 17:47 +0200, Michael S. Tsirkin wrote:
quoted
On Wed, Feb 02, 2011 at 07:39:45AM -0800, Shirley Ma wrote:
quoted
On Wed, 2011-02-02 at 12:48 +0200, Michael S. Tsirkin wrote:
quoted
Yes, I think doing this in the host is much simpler,
just send an interrupt after there's a decent amount
of space in the queue.
Having said that the simple heuristic that I coded
might be a bit too simple.
quoted
From the debugging out what I have seen so far (a single small
message
quoted
TCP_STEAM test), I think the right approach is to patch both guest
and
quoted
vhost.
One problem is slowing down the guest helps here.
So there's a chance that just by adding complexity
in guest driver we get a small improvement :(
We can't rely on a patched guest anyway, so
I think it is best to test guest and host changes separately.
And I do agree something needs to be done in guest too,
for example when vqs share an interrupt, we
might invoke a callback when we see vq is not empty
even though it's not requested. Probably should
check interrupts enabled here?
Yes, I modified xmit callback something like below:
static void skb_xmit_done(struct virtqueue *svq)
{
struct virtnet_info *vi = svq->vdev->priv;
/* Suppress further interrupts. */
virtqueue_disable_cb(svq);
/* We were probably waiting for more output buffers. */
if (netif_queue_stopped(vi->dev)) {
free_old_xmit_skbs(vi);
if (virtqueue_free_size(svq) <= svq->vring.num / 2) {
virtqueue_enable_cb(svq);
return;
}
}
netif_wake_queue(vi->dev);
}
OK, but this should have no effect with a vhost patch
which should ensure that we don't get an interrupt
until the queue is at least half empty.
Right?
quoted
quoted
The problem I have found is a regression for single small
message TCP_STEAM test. Old kernel works well for TCP_STREAM, only
new
quoted
kernel has problem.
Likely new kernel is faster :)
quoted
quoted
For Steven's problem, it's multiple stream TCP_RR issues, the old
guest
quoted
doesn't perform well, so does new guest kernel. We tested reducing
vhost
quoted
signaling patch before, it didn't help the performance at all.
Thanks
Shirley
Yes, it seems unrelated to tx interrupts.
The issue is more likely related to latency.
Could be. Why do you think so?
Do you have anything in
mind on how to reduce vhost latency?
Thanks
Shirley
Hmm, bypassing the bridge might help a bit.
Are you using tap+bridge or macvtap?
From: Shirley Ma <hidden> Date: 2011-02-02 18:11:51
On Wed, 2011-02-02 at 19:32 +0200, Michael S. Tsirkin wrote:
OK, but this should have no effect with a vhost patch
which should ensure that we don't get an interrupt
until the queue is at least half empty.
Right?
There should be some coordination between guest and vhost. We shouldn't
count the TX packets when netif queue is enabled since next guest TX
xmit will free any used buffers in vhost. We need to be careful here in
case we miss the interrupts when netif queue has stopped.
However we can't change old guest so we can test the patches separately
for guest only, vhost only, and the combination.
quoted
quoted
Yes, it seems unrelated to tx interrupts.
The issue is more likely related to latency.
Could be. Why do you think so?
Since I played with latency hack, I can see performance difference for
different latency.
quoted
Do you have anything in
mind on how to reduce vhost latency?
Thanks
Shirley
Hmm, bypassing the bridge might help a bit.
Are you using tap+bridge or macvtap?
I am using tap+bridge for TCP_RR test, I think Steven tested macvtap
before. He might have some data from his workload performance
measurement.
Shirley
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2011-02-02 18:20:52
On Wed, Feb 02, 2011 at 07:42:51AM -0800, Shirley Ma wrote:
On Wed, 2011-02-02 at 12:49 +0200, Michael S. Tsirkin wrote:
quoted
On Tue, Feb 01, 2011 at 11:33:49PM -0800, Shirley Ma wrote:
quoted
On Tue, 2011-02-01 at 23:14 -0800, Shirley Ma wrote:
quoted
w/i guest change, I played around the parameters,for example: I
could
quoted
quoted
get 3.7Gb/s with 42% CPU BW increasing from 2.5Gb/s for 1K message
size,
w/i dropping packet, I was able to get up to 6.2Gb/s with similar
CPU
quoted
quoted
usage.
I meant w/o guest change, only vhost changes. Sorry about that.
Shirley
Ah, excellent. What were the parameters?
I used half of the ring size 129 for packet counters,
but the
performance is still not as good as dropping packets on guest, 3.7 Gb/s
vs. 6.2Gb/s.
Shirley
How many packets and bytes per interrupt are sent?
Also, what about other values for the counters and other counters?
What does your patch do? Just drop packets instead of
stopping the interface?
To have an understanding when should we drop packets
in the guest, we need to know *why* does it help.
Otherwise, how do we know it will work for others?
Note that qdisc will drop packets when it overruns -
so what is different? Also, are we over-running some other queue
somewhere?
--
MST
From: Shirley Ma <hidden> Date: 2011-02-02 18:26:09
On Wed, 2011-02-02 at 20:20 +0200, Michael S. Tsirkin wrote:
How many packets and bytes per interrupt are sent?
Also, what about other values for the counters and other counters?
What does your patch do? Just drop packets instead of
stopping the interface?
To have an understanding when should we drop packets
in the guest, we need to know *why* does it help.
Otherwise, how do we know it will work for others?
Note that qdisc will drop packets when it overruns -
so what is different? Also, are we over-running some other queue
somewhere?
Agreed. I am trying to put more debugging output to look for all these
answers.
Shirley
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2011-02-02 18:27:48
On Wed, Feb 02, 2011 at 10:11:51AM -0800, Shirley Ma wrote:
On Wed, 2011-02-02 at 19:32 +0200, Michael S. Tsirkin wrote:
quoted
OK, but this should have no effect with a vhost patch
which should ensure that we don't get an interrupt
until the queue is at least half empty.
Right?
There should be some coordination between guest and vhost.
What kind of coordination? With a patched vhost, and a full ring.
you should get an interrupt per 100 packets.
Is this what you see? And if yes, isn't the guest patch
doing nothing then?
We shouldn't
count the TX packets when netif queue is enabled since next guest TX
xmit will free any used buffers in vhost. We need to be careful here in
case we miss the interrupts when netif queue has stopped.
However we can't change old guest so we can test the patches separately
for guest only, vhost only, and the combination.
quoted
quoted
quoted
Yes, it seems unrelated to tx interrupts.
The issue is more likely related to latency.
Could be. Why do you think so?
Since I played with latency hack, I can see performance difference for
different latency.
Which hack was that?
quoted
quoted
Do you have anything in
mind on how to reduce vhost latency?
Thanks
Shirley
Hmm, bypassing the bridge might help a bit.
Are you using tap+bridge or macvtap?
I am using tap+bridge for TCP_RR test, I think Steven tested macvtap
before. He might have some data from his workload performance
measurement.
Shirley
From: Shirley Ma <hidden> Date: 2011-02-02 19:29:43
On Wed, 2011-02-02 at 20:27 +0200, Michael S. Tsirkin wrote:
On Wed, Feb 02, 2011 at 10:11:51AM -0800, Shirley Ma wrote:
quoted
On Wed, 2011-02-02 at 19:32 +0200, Michael S. Tsirkin wrote:
quoted
OK, but this should have no effect with a vhost patch
which should ensure that we don't get an interrupt
until the queue is at least half empty.
Right?
There should be some coordination between guest and vhost.
What kind of coordination? With a patched vhost, and a full ring.
you should get an interrupt per 100 packets.
Is this what you see? And if yes, isn't the guest patch
doing nothing then?
vhost_signal won't be able send any TX interrupts to guest when guest TX
interrupt is disabled. Guest TX interrupt is only enabled when running
out of descriptors.
quoted
We shouldn't
count the TX packets when netif queue is enabled since next guest TX
xmit will free any used buffers in vhost. We need to be careful here
in
quoted
case we miss the interrupts when netif queue has stopped.
However we can't change old guest so we can test the patches
separately
quoted
for guest only, vhost only, and the combination.
quoted
quoted
quoted
Yes, it seems unrelated to tx interrupts.
The issue is more likely related to latency.
Could be. Why do you think so?
Since I played with latency hack, I can see performance difference
for
quoted
different latency.
Which hack was that?
I tried to accumulate multiple guest to host notifications for TX xmits,
it did help multiple streams TCP_RR results; I also forced vhost
handle_tx to handle more packets; both hack seemed help.
Thanks
Shirley
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2011-02-02 20:17:31
On Wed, Feb 02, 2011 at 11:29:35AM -0800, Shirley Ma wrote:
On Wed, 2011-02-02 at 20:27 +0200, Michael S. Tsirkin wrote:
quoted
On Wed, Feb 02, 2011 at 10:11:51AM -0800, Shirley Ma wrote:
quoted
On Wed, 2011-02-02 at 19:32 +0200, Michael S. Tsirkin wrote:
quoted
OK, but this should have no effect with a vhost patch
which should ensure that we don't get an interrupt
until the queue is at least half empty.
Right?
There should be some coordination between guest and vhost.
What kind of coordination? With a patched vhost, and a full ring.
you should get an interrupt per 100 packets.
Is this what you see? And if yes, isn't the guest patch
doing nothing then?
vhost_signal won't be able send any TX interrupts to guest when guest TX
interrupt is disabled. Guest TX interrupt is only enabled when running
out of descriptors.
Well, this is also the only case where the queue is stopped, no?
quoted
quoted
We shouldn't
count the TX packets when netif queue is enabled since next guest TX
xmit will free any used buffers in vhost. We need to be careful here
in
quoted
case we miss the interrupts when netif queue has stopped.
However we can't change old guest so we can test the patches
separately
quoted
for guest only, vhost only, and the combination.
quoted
quoted
quoted
Yes, it seems unrelated to tx interrupts.
The issue is more likely related to latency.
Could be. Why do you think so?
Since I played with latency hack, I can see performance difference
for
quoted
different latency.
Which hack was that?
I tried to accumulate multiple guest to host notifications for TX xmits,
it did help multiple streams TCP_RR results;
I don't see a point to delay used idx update, do you?
So delaying just signal seems better, right?
I also forced vhost
handle_tx to handle more packets; both hack seemed help.
Thanks
Shirley
Haven't noticed that part, how does your patch make it
handle more packets?
--
MST
From: Shirley Ma <hidden> Date: 2011-02-02 21:03:05
On Wed, 2011-02-02 at 22:17 +0200, Michael S. Tsirkin wrote:
Well, this is also the only case where the queue is stopped, no?
Yes. I got some debugging data, I saw that sometimes there were so many
packets were waiting for free in guest between vhost_signal & guest xmit
callback. Looks like the time spent too long from vhost_signal to guest
xmit callback?
quoted
I tried to accumulate multiple guest to host notifications for TX
xmits,
quoted
it did help multiple streams TCP_RR results;
I don't see a point to delay used idx update, do you?
It might cause per vhost handle_tx processed more packets.
So delaying just signal seems better, right?
I think I need to define the test matrix to collect data for TX xmit
from guest to host here for different tests.
Data to be collected:
---------------------
1. kvm_stat for VM, I/O exits
2. cpu utilization for both guest and host
3. cat /proc/interrupts on guest
4. packets rate from vhost handle_tx per loop
5. guest netif queue stop rate
6. how many packets are waiting for free between vhost signaling and
guest callback
7. performance results
Test
----
1. TCP_STREAM single stream test for 1K to 4K message size
2. TCP_RR (64 instance test): 128 - 1K request/response size
Different hacks
---------------
1. Base line data ( with the patch to fix capacity check first,
free_old_xmit_skbs returns number of skbs)
2. Drop packet data (will put some debugging in generic networking code)
3. Delay guest netif queue wake up until certain descriptors (1/2 ring
size, 1/4 ring size...) are available once the queue has stopped.
4. Accumulate more packets per vhost signal in handle_tx?
5. 3 & 4 combinations
6. Accumulate more packets per guest kick() (TCP_RR) by adding a timer?
7. Accumulate more packets per vhost handle_tx() by adding some delay?
Haven't noticed that part, how does your patch make it
handle more packets?
Added a delay in handle_tx().
What else?
It would take sometimes to do this.
Shirley
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2011-02-02 21:20:47
On Wed, Feb 02, 2011 at 01:03:05PM -0800, Shirley Ma wrote:
On Wed, 2011-02-02 at 22:17 +0200, Michael S. Tsirkin wrote:
quoted
Well, this is also the only case where the queue is stopped, no?
Yes. I got some debugging data, I saw that sometimes there were so many
packets were waiting for free in guest between vhost_signal & guest xmit
callback.
What does this mean?
Looks like the time spent too long from vhost_signal to guest
xmit callback?
quoted
quoted
I tried to accumulate multiple guest to host notifications for TX
xmits,
quoted
it did help multiple streams TCP_RR results;
I don't see a point to delay used idx update, do you?
It might cause per vhost handle_tx processed more packets.
I don't understand. It's a couple of writes - what is the issue?
quoted
So delaying just signal seems better, right?
I think I need to define the test matrix to collect data for TX xmit
from guest to host here for different tests.
Data to be collected:
---------------------
1. kvm_stat for VM, I/O exits
2. cpu utilization for both guest and host
3. cat /proc/interrupts on guest
4. packets rate from vhost handle_tx per loop
5. guest netif queue stop rate
6. how many packets are waiting for free between vhost signaling and
guest callback
7. performance results
Test
----
1. TCP_STREAM single stream test for 1K to 4K message size
2. TCP_RR (64 instance test): 128 - 1K request/response size
Different hacks
---------------
1. Base line data ( with the patch to fix capacity check first,
free_old_xmit_skbs returns number of skbs)
2. Drop packet data (will put some debugging in generic networking code)
3. Delay guest netif queue wake up until certain descriptors (1/2 ring
size, 1/4 ring size...) are available once the queue has stopped.
4. Accumulate more packets per vhost signal in handle_tx?
5. 3 & 4 combinations
6. Accumulate more packets per guest kick() (TCP_RR) by adding a timer?
7. Accumulate more packets per vhost handle_tx() by adding some delay?
quoted
Haven't noticed that part, how does your patch make it
handle more packets?
Added a delay in handle_tx().
What else?
It would take sometimes to do this.
Shirley
From: Shirley Ma <hidden> Date: 2011-02-02 21:41:39
On Wed, 2011-02-02 at 23:20 +0200, Michael S. Tsirkin wrote:
quoted
On Wed, 2011-02-02 at 22:17 +0200, Michael S. Tsirkin wrote:
quoted
Well, this is also the only case where the queue is stopped, no?
Yes. I got some debugging data, I saw that sometimes there were so
many
quoted
packets were waiting for free in guest between vhost_signal & guest
xmit
quoted
callback.
What does this mean?
Let's look at the sequence here:
guest start_xmit()
xmit_skb()
if ring is full,
enable_cb()
guest skb_xmit_done()
disable_cb,
printk free_old_xmit_skbs <-- it was between more than 1/2 to
full ring size
printk vq->num_free
vhost handle_tx()
if (guest interrupt is enabled)
signal guest to free xmit buffers
So between guest queue full/stopped queue/enable call back to guest
receives the callback from host to free_old_xmit_skbs, there were about
1/2 to full ring size descriptors available. I thought there were only a
few. (I disabled your vhost patch for this test.)
quoted
Looks like the time spent too long from vhost_signal to guest
xmit callback?
quoted
quoted
quoted
I tried to accumulate multiple guest to host notifications for
TX
quoted
quoted
xmits,
quoted
it did help multiple streams TCP_RR results;
I don't see a point to delay used idx update, do you?
It might cause per vhost handle_tx processed more packets.
I don't understand. It's a couple of writes - what is the issue?
Oh, handle_tx could process more packets per loop for multiple streams
TCP_RR case. I need to print out the data rate per loop to confirm this.
Shirley
From: Shirley Ma <hidden> Date: 2011-02-03 05:05:56
On Wed, 2011-02-02 at 23:20 +0200, Michael S. Tsirkin wrote:
quoted
I think I need to define the test matrix to collect data for TX xmit
from guest to host here for different tests.
Data to be collected:
---------------------
1. kvm_stat for VM, I/O exits
2. cpu utilization for both guest and host
3. cat /proc/interrupts on guest
4. packets rate from vhost handle_tx per loop
5. guest netif queue stop rate
6. how many packets are waiting for free between vhost signaling and
guest callback
7. performance results
Test
----
1. TCP_STREAM single stream test for 1K to 4K message size
2. TCP_RR (64 instance test): 128 - 1K request/response size
Different hacks
---------------
1. Base line data ( with the patch to fix capacity check first,
free_old_xmit_skbs returns number of skbs)
2. Drop packet data (will put some debugging in generic networking
code)
Since I found that the netif queue stop/wake up is so expensive, I
created a dropping packets patch on guest side so I don't need to debug
generic networking code.
guest start_xmit()
capacity = free_old_xmit_skb() + virtqueue_get_num_freed()
if (capacity == 0)
drop this packet;
return;
In the patch, both guest TX interrupts and callback have been omitted.
Host vhost_signal in handle_tx can totally be removed as well. (A new
virtio_ring API is needed for exporting total of num_free descriptors
here -- virtioqueue_get_num_freed)
Initial TCP_STREAM performance results I got for guest to local host
4.2Gb/s for 1K message size, (vs. 2.5Gb/s)
6.2Gb/s for 2K message size, and (vs. 3.8Gb/s)
9.8Gb/s for 4K message size. (vs.5.xGb/s)
Since large message size (64K) doesn't hit (capacity == 0) case, so the
performance only has a little better. (from 13.xGb/s to 14.x Gb/s)
kvm_stat output shows significant exits reduction for both VM and I/O,
no guest TX interrupts.
With dropping packets, TCP retrans has been increased here, so I can see
performance numbers are various.
This might be not a good solution, but it gave us some ideas on
expensive netif queue stop/wake up between guest and host notification.
I couldn't find a better solution on how to reduce netif queue stop/wake
up rate for small message size. But I think once we can address this,
the guest TX performance will burst for small message size.
I also compared this with return TX_BUSY approach when (capacity == 0),
it is not as good as dropping packets.
quoted
3. Delay guest netif queue wake up until certain descriptors (1/2
ring
quoted
size, 1/4 ring size...) are available once the queue has stopped.
4. Accumulate more packets per vhost signal in handle_tx?
5. 3 & 4 combinations
6. Accumulate more packets per guest kick() (TCP_RR) by adding a
timer?
quoted
7. Accumulate more packets per vhost handle_tx() by adding some
delay?
quoted
quoted
Haven't noticed that part, how does your patch make it
handle more packets?
Added a delay in handle_tx().
What else?
It would take sometimes to do this.
Shirley
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2011-02-03 05:59:46
On Wed, Feb 02, 2011 at 01:41:33PM -0800, Shirley Ma wrote:
On Wed, 2011-02-02 at 23:20 +0200, Michael S. Tsirkin wrote:
quoted
quoted
On Wed, 2011-02-02 at 22:17 +0200, Michael S. Tsirkin wrote:
quoted
Well, this is also the only case where the queue is stopped, no?
Yes. I got some debugging data, I saw that sometimes there were so
many
quoted
packets were waiting for free in guest between vhost_signal & guest
xmit
quoted
callback.
What does this mean?
Let's look at the sequence here:
guest start_xmit()
xmit_skb()
if ring is full,
enable_cb()
guest skb_xmit_done()
disable_cb,
printk free_old_xmit_skbs <-- it was between more than 1/2 to
full ring size
printk vq->num_free
vhost handle_tx()
if (guest interrupt is enabled)
signal guest to free xmit buffers
So between guest queue full/stopped queue/enable call back to guest
receives the callback from host to free_old_xmit_skbs, there were about
1/2 to full ring size descriptors available. I thought there were only a
few. (I disabled your vhost patch for this test.)
The expected number is vq->num - max skb frags - 2.
quoted
quoted
Looks like the time spent too long from vhost_signal to guest
xmit callback?
quoted
quoted
quoted
I tried to accumulate multiple guest to host notifications for
TX
quoted
quoted
xmits,
quoted
it did help multiple streams TCP_RR results;
I don't see a point to delay used idx update, do you?
It might cause per vhost handle_tx processed more packets.
I don't understand. It's a couple of writes - what is the issue?
Oh, handle_tx could process more packets per loop for multiple streams
TCP_RR case. I need to print out the data rate per loop to confirm this.
Shirley
From: Shirley Ma <hidden> Date: 2011-02-03 06:09:14
On Thu, 2011-02-03 at 07:59 +0200, Michael S. Tsirkin wrote:
quoted
Let's look at the sequence here:
guest start_xmit()
xmit_skb()
if ring is full,
enable_cb()
guest skb_xmit_done()
disable_cb,
printk free_old_xmit_skbs <-- it was between more than 1/2
to
quoted
full ring size
printk vq->num_free
vhost handle_tx()
if (guest interrupt is enabled)
signal guest to free xmit buffers
So between guest queue full/stopped queue/enable call back to guest
receives the callback from host to free_old_xmit_skbs, there were
about
quoted
1/2 to full ring size descriptors available. I thought there were
only a
quoted
few. (I disabled your vhost patch for this test.)
The expected number is vq->num - max skb frags - 2.
It was various (up to the ring size 256). This is using indirection
buffers, it returned how many freed descriptors, not number of buffers.
Why do you think it is vq->num - max skb frags - 2 here?
Shirley
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2011-02-03 06:13:47
On Wed, Feb 02, 2011 at 09:05:56PM -0800, Shirley Ma wrote:
On Wed, 2011-02-02 at 23:20 +0200, Michael S. Tsirkin wrote:
quoted
quoted
I think I need to define the test matrix to collect data for TX xmit
from guest to host here for different tests.
Data to be collected:
---------------------
1. kvm_stat for VM, I/O exits
2. cpu utilization for both guest and host
3. cat /proc/interrupts on guest
4. packets rate from vhost handle_tx per loop
5. guest netif queue stop rate
6. how many packets are waiting for free between vhost signaling and
guest callback
7. performance results
Test
----
1. TCP_STREAM single stream test for 1K to 4K message size
2. TCP_RR (64 instance test): 128 - 1K request/response size
Different hacks
---------------
1. Base line data ( with the patch to fix capacity check first,
free_old_xmit_skbs returns number of skbs)
2. Drop packet data (will put some debugging in generic networking
code)
Since I found that the netif queue stop/wake up is so expensive, I
created a dropping packets patch on guest side so I don't need to debug
generic networking code.
guest start_xmit()
capacity = free_old_xmit_skb() + virtqueue_get_num_freed()
if (capacity == 0)
drop this packet;
return;
In the patch, both guest TX interrupts and callback have been omitted.
Host vhost_signal in handle_tx can totally be removed as well. (A new
virtio_ring API is needed for exporting total of num_free descriptors
here -- virtioqueue_get_num_freed)
Initial TCP_STREAM performance results I got for guest to local host
4.2Gb/s for 1K message size, (vs. 2.5Gb/s)
6.2Gb/s for 2K message size, and (vs. 3.8Gb/s)
9.8Gb/s for 4K message size. (vs.5.xGb/s)
What is the average packet size, # bytes per ack, and the # of interrupts
per packet? It could be that just slowing down trahsmission
makes GSO work better.
Since large message size (64K) doesn't hit (capacity == 0) case, so the
performance only has a little better. (from 13.xGb/s to 14.x Gb/s)
kvm_stat output shows significant exits reduction for both VM and I/O,
no guest TX interrupts.
With dropping packets, TCP retrans has been increased here, so I can see
performance numbers are various.
This might be not a good solution, but it gave us some ideas on
expensive netif queue stop/wake up between guest and host notification.
I couldn't find a better solution on how to reduce netif queue stop/wake
up rate for small message size. But I think once we can address this,
the guest TX performance will burst for small message size.
I also compared this with return TX_BUSY approach when (capacity == 0),
it is not as good as dropping packets.
quoted
quoted
3. Delay guest netif queue wake up until certain descriptors (1/2
ring
quoted
size, 1/4 ring size...) are available once the queue has stopped.
4. Accumulate more packets per vhost signal in handle_tx?
5. 3 & 4 combinations
6. Accumulate more packets per guest kick() (TCP_RR) by adding a
timer?
quoted
7. Accumulate more packets per vhost handle_tx() by adding some
delay?
quoted
quoted
Haven't noticed that part, how does your patch make it
handle more packets?
Added a delay in handle_tx().
What else?
It would take sometimes to do this.
Shirley
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2011-02-03 06:16:29
On Wed, Feb 02, 2011 at 10:09:14PM -0800, Shirley Ma wrote:
On Thu, 2011-02-03 at 07:59 +0200, Michael S. Tsirkin wrote:
quoted
quoted
Let's look at the sequence here:
guest start_xmit()
xmit_skb()
if ring is full,
enable_cb()
guest skb_xmit_done()
disable_cb,
printk free_old_xmit_skbs <-- it was between more than 1/2
to
quoted
full ring size
printk vq->num_free
vhost handle_tx()
if (guest interrupt is enabled)
signal guest to free xmit buffers
So between guest queue full/stopped queue/enable call back to guest
receives the callback from host to free_old_xmit_skbs, there were
about
quoted
1/2 to full ring size descriptors available. I thought there were
only a
quoted
few. (I disabled your vhost patch for this test.)
The expected number is vq->num - max skb frags - 2.
It was various (up to the ring size 256). This is using indirection
buffers, it returned how many freed descriptors, not number of buffers.
Why do you think it is vq->num - max skb frags - 2 here?
Shirley
well queue is stopped which happens when
if (capacity < 2+MAX_SKB_FRAGS) {
netif_stop_queue(dev);
if (unlikely(!virtqueue_enable_cb(vi->svq))) {
/* More just got used, free them then recheck.
* */
capacity += free_old_xmit_skbs(vi);
if (capacity >= 2+MAX_SKB_FRAGS) {
netif_start_queue(dev);
virtqueue_disable_cb(vi->svq);
}
}
}
This should be the most common case.
I guess the case with += free_old_xmit_skbs is what can get us more.
But it should be rare. Can you count how common it is?
--
MST
From: Shirley Ma <hidden> Date: 2011-02-03 15:58:00
On Thu, 2011-02-03 at 08:13 +0200, Michael S. Tsirkin wrote:
quoted
Initial TCP_STREAM performance results I got for guest to local
host
quoted
4.2Gb/s for 1K message size, (vs. 2.5Gb/s)
6.2Gb/s for 2K message size, and (vs. 3.8Gb/s)
9.8Gb/s for 4K message size. (vs.5.xGb/s)
What is the average packet size, # bytes per ack, and the # of
interrupts
per packet? It could be that just slowing down trahsmission
makes GSO work better.
There is no TX interrupts with dropping packet.
GSO/TSO is the key for small message performance, w/o GSO/TSO, the
performance is limited to about 2Gb/s no matter how big the message size
it is. I think any work we try here will increase large packet size
rate. BTW for dropping packet, TCP increased fast retrans, not slow
start.
I will collect tcpdump, netstart before and after data to compare packet
size/rate w/o w/i the patch.
Thanks
Shirley
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2011-02-03 16:21:06
On Thu, Feb 03, 2011 at 07:58:00AM -0800, Shirley Ma wrote:
On Thu, 2011-02-03 at 08:13 +0200, Michael S. Tsirkin wrote:
quoted
quoted
Initial TCP_STREAM performance results I got for guest to local
host
quoted
4.2Gb/s for 1K message size, (vs. 2.5Gb/s)
6.2Gb/s for 2K message size, and (vs. 3.8Gb/s)
9.8Gb/s for 4K message size. (vs.5.xGb/s)
What is the average packet size, # bytes per ack, and the # of
interrupts
per packet? It could be that just slowing down trahsmission
makes GSO work better.
There is no TX interrupts with dropping packet.
GSO/TSO is the key for small message performance, w/o GSO/TSO, the
performance is limited to about 2Gb/s no matter how big the message size
it is. I think any work we try here will increase large packet size
rate. BTW for dropping packet, TCP increased fast retrans, not slow
start.
I will collect tcpdump, netstart before and after data to compare packet
size/rate w/o w/i the patch.
Thanks
Shirley
Just a thought: does it help to make tx queue len of the
virtio device smaller?
E.g. match the vq size?
--
MST
From: Shirley Ma <hidden> Date: 2011-02-03 17:18:36
On Thu, 2011-02-03 at 18:20 +0200, Michael S. Tsirkin wrote:
Just a thought: does it help to make tx queue len of the
virtio device smaller?
Yes, that what I did before, reducing txqueuelen will cause qdisc dropp
the packet early, But it's hard to control by using tx queuelen for
performance gain. I tried on different systems, it required different
values.
Also, I tried another patch, instead of dropping packets, I used to
timer (2 jiffies) to enable/disable queue on guest without interrupts
notification, it gets better performance than original but worse
performance than dropping packets because of netif stop/wake up too
often.
vhost is definitely needed to improve for handling small message sizes.
It's unable to handle small message packets rate for queue size 256,
even with ring size 1024. QEMU seems not allowing to increase the TX
ring size to 2K (start qemu_kvm failure with no errors), I am not able
to test it out.
Thanks
Shirley