From: Paolo Valente <hidden> Date: 2021-02-05 07:50:19
Il giorno 29 gen 2021, alle ore 09:28, yukuai (C) [off-list ref] ha scritto:
Hi,
Thanks for your response, and my apologize for the delay, my tmie
is very limited recently.
I do know that problem ...
On 2021/01/22 18:09, Paolo Valente wrote:
quoted
Hi,
this is a core problem, not of BFQ but of any possible solution that
has to provide bandwidth isolation with sync I/O. One of the examples
I'm not sure about this, so I test it with iocost in mq and cfq in sq,
result shows that they do can provide bandwidth isolation with sync I/O
without significant performance degradation.
Yep, that means just that, with your specific workload, bandwidth
isolation gets guaranteed without idling. So that's exactly one of
the workloads for which I'm suggesting my handling of a special case.
quoted
is the one I made for you in my other email. At any rate, the problem
that you report seems to occur with just one group. We may think of
simply changing my condition
bfqd->num_groups_with_pending_reqs > 0
to
bfqd->num_groups_with_pending_reqs > 1
We aredy tried this, the problem will dispeare if only one group is
active. And I think this modification is reasonable because
bandwidth isolation is not necessary in this case.
Thanks for your feedback. I'll consider submitting this change.
However, considering the common case, when more than one
group is active, and one of the group is issuing sync IO, I think
we need to find a way to prevent the preformance degradation.
I agree. What do you think of my suggestion for solving the problem?
Might you help with that?
Thanks,
Paolo
quoted
If this simple solution does solve the problem you report, then I
could run my batch of tests to check whether it causes some
regression.
What do you think?
Thanks.
Paolo
Il giorno 29 gen 2021, alle ore 09:28, yukuai (C) [off-list ref] ha scritto:
Hi,
Thanks for your response, and my apologize for the delay, my tmie
is very limited recently.
I do know that problem ...
quoted
On 2021/01/22 18:09, Paolo Valente wrote:
quoted
Hi,
this is a core problem, not of BFQ but of any possible solution that
has to provide bandwidth isolation with sync I/O. One of the examples
I'm not sure about this, so I test it with iocost in mq and cfq in sq,
result shows that they do can provide bandwidth isolation with sync I/O
without significant performance degradation.
Yep, that means just that, with your specific workload, bandwidth
isolation gets guaranteed without idling. So that's exactly one of
the workloads for which I'm suggesting my handling of a special case.
quoted
quoted
is the one I made for you in my other email. At any rate, the problem
that you report seems to occur with just one group. We may think of
simply changing my condition
bfqd->num_groups_with_pending_reqs > 0
to
bfqd->num_groups_with_pending_reqs > 1
We aredy tried this, the problem will dispeare if only one group is
active. And I think this modification is reasonable because
bandwidth isolation is not necessary in this case.
Thanks for your feedback. I'll consider submitting this change.
quoted
However, considering the common case, when more than one
group is active, and one of the group is issuing sync IO, I think
we need to find a way to prevent the preformance degradation.
I agree. What do you think of my suggestion for solving the problem?
Might you help with that?
Hi
Do you mead the suggestion that you mentioned in another email:
"a varied_rq_size flag, similar to the varied_weights flag" ?
I'm afraid that's just a circumvention plan, not a solution to the
special case.
By the way, I'm glad if there is anything I can help, however it'll
wait for a few days cause the Spring Festival is coming.
Thanks,
Yu Kuai
Thanks,
Paolo
quoted
quoted
If this simple solution does solve the problem you report, then I
could run my batch of tests to check whether it causes some
regression.
What do you think?
Thanks.
Paolo
From: Paolo Valente <hidden> Date: 2021-02-08 19:09:01
Il giorno 7 feb 2021, alle ore 13:49, yukuai (C) [off-list ref] ha scritto:
On 2021/02/05 15:49, Paolo Valente wrote:
quoted
quoted
Il giorno 29 gen 2021, alle ore 09:28, yukuai (C) [off-list ref] ha scritto:
Hi,
Thanks for your response, and my apologize for the delay, my tmie
is very limited recently.
I do know that problem ...
quoted
On 2021/01/22 18:09, Paolo Valente wrote:
quoted
Hi,
this is a core problem, not of BFQ but of any possible solution that
has to provide bandwidth isolation with sync I/O. One of the examples
I'm not sure about this, so I test it with iocost in mq and cfq in sq,
result shows that they do can provide bandwidth isolation with sync I/O
without significant performance degradation.
Yep, that means just that, with your specific workload, bandwidth
isolation gets guaranteed without idling. So that's exactly one of
the workloads for which I'm suggesting my handling of a special case.
quoted
quoted
is the one I made for you in my other email. At any rate, the problem
that you report seems to occur with just one group. We may think of
simply changing my condition
bfqd->num_groups_with_pending_reqs > 0
to
bfqd->num_groups_with_pending_reqs > 1
We aredy tried this, the problem will dispeare if only one group is
active. And I think this modification is reasonable because
bandwidth isolation is not necessary in this case.
Thanks for your feedback. I'll consider submitting this change.
quoted
However, considering the common case, when more than one
group is active, and one of the group is issuing sync IO, I think
we need to find a way to prevent the preformance degradation.
I agree. What do you think of my suggestion for solving the problem?
Might you help with that?
Hi
Do you mead the suggestion that you mentioned in another email:
"a varied_rq_size flag, similar to the varied_weights flag" ?
I'm afraid that's just a circumvention plan, not a solution to the
special case.
I'm a little confused. Could you explain why you think this is a
circumvention plan? Maybe even better, could you describe in detail
the special case you have in mind? We could start from there, to think
of a possible, satisfactory solution.
By the way, I'm glad if there is anything I can help, however it'll
wait for a few days cause the Spring Festival is coming.
Ok, Happy Spring Festival then.
Thanks.
Paolo
Thanks,
Yu Kuai
quoted
Thanks,
Paolo
quoted
quoted
If this simple solution does solve the problem you report, then I
could run my batch of tests to check whether it causes some
regression.
What do you think?
Thanks.
Paolo
Il giorno 7 feb 2021, alle ore 13:49, yukuai (C) [off-list ref] ha scritto:
On 2021/02/05 15:49, Paolo Valente wrote:
quoted
quoted
Il giorno 29 gen 2021, alle ore 09:28, yukuai (C) [off-list ref] ha scritto:
Hi,
Thanks for your response, and my apologize for the delay, my tmie
is very limited recently.
I do know that problem ...
quoted
On 2021/01/22 18:09, Paolo Valente wrote:
quoted
Hi,
this is a core problem, not of BFQ but of any possible solution that
has to provide bandwidth isolation with sync I/O. One of the examples
I'm not sure about this, so I test it with iocost in mq and cfq in sq,
result shows that they do can provide bandwidth isolation with sync I/O
without significant performance degradation.
Yep, that means just that, with your specific workload, bandwidth
isolation gets guaranteed without idling. So that's exactly one of
the workloads for which I'm suggesting my handling of a special case.
quoted
quoted
is the one I made for you in my other email. At any rate, the problem
that you report seems to occur with just one group. We may think of
simply changing my condition
bfqd->num_groups_with_pending_reqs > 0
to
bfqd->num_groups_with_pending_reqs > 1
We aredy tried this, the problem will dispeare if only one group is
active. And I think this modification is reasonable because
bandwidth isolation is not necessary in this case.
Thanks for your feedback. I'll consider submitting this change.
quoted
However, considering the common case, when more than one
group is active, and one of the group is issuing sync IO, I think
we need to find a way to prevent the preformance degradation.
I agree. What do you think of my suggestion for solving the problem?
Might you help with that?
Hi
Do you mead the suggestion that you mentioned in another email:
"a varied_rq_size flag, similar to the varied_weights flag" ?
I'm afraid that's just a circumvention plan, not a solution to the
special case.
I'm a little confused. Could you explain why you think this is a
circumvention plan? Maybe even better, could you describe in detail
the special case you have in mind? We could start from there, to think
of a possible, satisfactory solution.
Hi,
First of all, there are two conditions to trigger the problem in bfq:
a. issuing sync IO concurrently. (I was testing in one cgroup, and
I think multiple cgroups is the same.)
b. not issuing in root cgroup.
The phenomenon is that the performance will degradated to single
process. The reason is that bfq_queue will never expired untill
BUDGET_TIMEOUT since num_groups_with_pending_reqs will always be
nonzero after issuing io to driver, which means that there is only
one request in progress(during D2C) at any given moment.
I was trying to skip the checking of active group if bfq.weight is not
changed, and it's implemented by adding a varible 'check_active_group',
it'll only set to true if bfq.weight is changed.
This approach will work fine is the weight stay unchanged, and will
not be effective if the weight ever changed. This is why I said it's
a circumvention plan.
By the way, while testing with cfq, I found that the current active cfq
queue can be preempted in such special case. I wonder if it's worth
referring to bfq.
Thanks,
Yu Kuai
quoted
By the way, I'm glad if there is anything I can help, however it'll
wait for a few days cause the Spring Festival is coming.
Ok, Happy Spring Festival then.
Thanks.
Paolo
quoted
Thanks,
Yu Kuai
quoted
Thanks,
Paolo
quoted
quoted
If this simple solution does solve the problem you report, then I
could run my batch of tests to check whether it causes some
regression.
What do you think?
Thanks.
Paolo
From: Paolo Valente <hidden> Date: 2021-02-23 10:52:31
Il giorno 19 feb 2021, alle ore 13:03, yukuai (C) [off-list ref] ha scritto:
On 2021/02/09 3:05, Paolo Valente wrote:
quoted
quoted
Il giorno 7 feb 2021, alle ore 13:49, yukuai (C) [off-list ref] ha scritto:
On 2021/02/05 15:49, Paolo Valente wrote:
quoted
quoted
Il giorno 29 gen 2021, alle ore 09:28, yukuai (C) [off-list ref] ha scritto:
Hi,
Thanks for your response, and my apologize for the delay, my tmie
is very limited recently.
I do know that problem ...
quoted
On 2021/01/22 18:09, Paolo Valente wrote:
quoted
Hi,
this is a core problem, not of BFQ but of any possible solution that
has to provide bandwidth isolation with sync I/O. One of the examples
I'm not sure about this, so I test it with iocost in mq and cfq in sq,
result shows that they do can provide bandwidth isolation with sync I/O
without significant performance degradation.
Yep, that means just that, with your specific workload, bandwidth
isolation gets guaranteed without idling. So that's exactly one of
the workloads for which I'm suggesting my handling of a special case.
quoted
quoted
is the one I made for you in my other email. At any rate, the problem
that you report seems to occur with just one group. We may think of
simply changing my condition
bfqd->num_groups_with_pending_reqs > 0
to
bfqd->num_groups_with_pending_reqs > 1
We aredy tried this, the problem will dispeare if only one group is
active. And I think this modification is reasonable because
bandwidth isolation is not necessary in this case.
Thanks for your feedback. I'll consider submitting this change.
quoted
However, considering the common case, when more than one
group is active, and one of the group is issuing sync IO, I think
we need to find a way to prevent the preformance degradation.
I agree. What do you think of my suggestion for solving the problem?
Might you help with that?
Hi
Do you mead the suggestion that you mentioned in another email:
"a varied_rq_size flag, similar to the varied_weights flag" ?
I'm afraid that's just a circumvention plan, not a solution to the
special case.
I'm a little confused. Could you explain why you think this is a
circumvention plan? Maybe even better, could you describe in detail
the special case you have in mind? We could start from there, to think
of a possible, satisfactory solution.
Hi,
First of all, there are two conditions to trigger the problem in bfq:
a. issuing sync IO concurrently. (I was testing in one cgroup, and
I think multiple cgroups is the same.)
b. not issuing in root cgroup.
The phenomenon is that the performance will degradated to single
process. The reason is that bfq_queue will never expired untill
BUDGET_TIMEOUT since num_groups_with_pending_reqs will always be
nonzero after issuing io to driver, which means that there is only
one request in progress(during D2C) at any given moment.
I was trying to skip the checking of active group if bfq.weight is not
changed, and it's implemented by adding a varible 'check_active_group',
it'll only set to true if bfq.weight is changed.
This approach will work fine is the weight stay unchanged, and will
not be effective if the weight ever changed. This is why I said it's
a circumvention plan.
By the way, while testing with cfq, I found that the current active cfq
queue can be preempted in such special case. I wonder if it's worth
referring to bfq.
Hi,
thank you very much for this information.
You confirm my suspect: your case is one of those for which idling is
not needed. It only kills throughput.
Unfortunately, the preemption mechanism of cfq that you cite kills
service guaranteed in asymmetric cases. That's why I have never added
it to bfq.
Turning to solutions, now I also understand why you speak about a
circumvention plan. Yet the solution I described is not yours, and is
effective with any history of weight changes.
In a sense, my solution is an evolution of your initial solution. It
is conceptually very simple: just track whether all weights and all
I/O sizes are equal. When this condition holds true, no idling is
needed. When it does not hold, idling is needed. Your case would be
solved, service guarantees would be always preserved.
Thanks.
Paolo
Thanks,
Yu Kuai
quoted
quoted
By the way, I'm glad if there is anything I can help, however it'll
wait for a few days cause the Spring Festival is coming.
Ok, Happy Spring Festival then.
Thanks.
Paolo
quoted
Thanks,
Yu Kuai
quoted
Thanks,
Paolo
quoted
quoted
If this simple solution does solve the problem you report, then I
could run my batch of tests to check whether it causes some
regression.
What do you think?
Thanks.
Paolo
In a sense, my solution is an evolution of your initial solution. It
is conceptually very simple: just track whether all weights and all
I/O sizes are equal. When this condition holds true, no idling is
needed. When it does not hold, idling is needed. Your case would be
solved, service guarantees would be always preserved.
Hi,
The solution sounds good, and it's ture that my case would be solved.
Thank you very much!
Yu Kuai