Hello guys,
I'm new to livepatch world and now I'm a little confused with livepatch's
per-task consistency model which is introduced by [1]. I've also readed the
discussion on mailing list [2], which introduces shadow variable to handle
data layout and semantic changes. But there's still some confusion with this
per-task consistency model.
According to the model, there will be scenario where old function and new
function can co-exist, though for a single thread, it sees either all new
functions or all old functions.
I can't understand why Vojtech said that 'old func processing new data' was
impossible. Assuming a scenario where a process calls func-A to submit a
work request (inserted into a global list), and then a kthread is
responsible
for calling func-B to process all work requests in the list. What if
this process
has finished the transition (sees new func-A) while kthread still sees
the old func-B?
In this case, old func-B has to process new data. If there's some lock
semantic
changes in func-A and func-B, then old func-B has no way identifying the
shadow
variable labeled by new func-A.
Please tell me if I missed something, and any suggestions will be
appreciated. ;)
Thanks.
[1] livepatch: change to a per-task consistency model
(https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.6-rc3&id=d83a7cb375eec21f04c83542395d08b2f6641da2)
[2] https://lkml.kernel.org/r/20141107140458.GA21774@suse.cz
On Mon 2020-03-02 16:45:24, JeffleXu wrote:
Hello guys,
I'm new to livepatch world and now I'm a little confused with livepatch's
per-task consistency model which is introduced by [1]. I've also readed the
discussion on mailing list [2], which introduces shadow variable to handle
data layout and semantic changes. But there's still some confusion with this
per-task consistency model.
According to the model, there will be scenario where old function and new
function can co-exist, though for a single thread, it sees either all new
functions or all old functions.
I can't understand why Vojtech said that 'old func processing new data' was
impossible. Assuming a scenario where a process calls func-A to submit a
work request (inserted into a global list), and then a kthread is
responsible
for calling func-B to process all work requests in the list. What if this
process
has finished the transition (sees new func-A) while kthread still sees the
old func-B?
In this case, old func-B has to process new data. If there's some lock
semantic
changes in func-A and func-B, then old func-B has no way identifying the
shadow
variable labeled by new func-A.
Please tell me if I missed something, and any suggestions will be
appreciated. ;)
No, you did not miss anything. The consistency is only per-thread,
If a livepatch is changing semantic of a global variable it must
allow doing the changes only when the entire system is using
the new code. And the new code must be able to deal with both
old and new data.
The new data semantic can be enable by post-patch callback
that is called when the transition has finished.
Fortunately, semantic changes are very rare at least in security
fixes.
Best Regards,
Petr
Hi,
JeffleXu [off-list ref] writes:
According to the model, there will be scenario where old function and new
function can co-exist, though for a single thread, it sees either all new
functions or all old functions.
That's correct.
I can't understand why Vojtech said that 'old func processing new
data' was impossible.
Just to make it explicit: Vojtech was talking about data layout
changes.
That is, consider you have something like e.g. this in the unmodified
kernel sources:
struct my_driver_work
{
struct work_struct work;
struct list_head works_list;
void *some_payload;
};
In general, you can't change that struct definition from a live
patch. So simply extending it like this
struct my_driver_work
{
struct work_struct work;
struct list_head works_list;
unsigned long flags /* New */
void *some_payload;
};
won't work.
Assuming a scenario where a process calls func-A to submit a
work request (inserted into a global list), and then a kthread is
responsible
for calling func-B to process all work requests in the list. What if
this process
has finished the transition (sees new func-A) while kthread still sees
the old func-B?
Going with the example from above, the patched func-A would submit
instances of the new struct my_driver_work whereas the unpatched func-B
would expect the ->some_payload pointer at where ->flags is stored now,
which is bad, obviously.
In this specific example, it could perhaps be possible to make the
patched func-A associate a shadow variable corresponding to the new
->flags member with newly created struct my_driver_work instances (of
original, unmodified layout). Any unpatched func-B would obviously
ignore it and consider the shadow only when it becomes patched. It very
much depends on the specific situation whether or not this is
acceptable. If not, the ->post_patch() can sometimes be used to achieve
some notion of a "global consistency" state (in this context,
c.f. Documentation/livepatch/system-state.rst).
Note however, that the patched func-B must always be able to handle
the situation where a struct my_driver_work instance does not have such
a ->flags shadow attached to it, either because the instance had been
created when the live patch has not been applied at all or because it
has been submitted from a not yet transitioned func-A.
There's another subtlety: the deallocation code for struct
my_driver_work needs to get livepatched as well to make it free the
->flags shadow. Consider the case where func-A has been transitioned,
but the deallocation code hasn't yet. Without any extra measures in
func-A, it could happen that a stale ->flags shadow from a deallocated
struct my_driver_work gets (wrongly) reassociated with a fresh struct
my_driver_work instance allocated at the same address as the old one
(because shadow variables are keyed on addresses of the data they're
associated with). Sometimes that's acceptable, sometimes it's not. In
the latter case you probably had to check for this situation and work
around it in the allocation code, i.e. the live-patched func-A.
Finally, let me remark that from my experience, most CVEs (>95%) can be
fixed via live patching without having to resort to either of shadow
variables, callbacks or the state API. For the rest, things usually tend
to become really non-trivial, hackish and subtle.
In this case, old func-B has to process new data. If there's some lock
semantic
changes in func-A and func-B, then old func-B has no way identifying
the shadow
variable labeled by new func-A.
I don't understand what you mean by "variable labeled by new func-A"?
Anyway, it's correct that an unpatched func-B would not consider any
shadow variables instantiated by patched func-A. And it's also correct
that changing locking semantics is difficult, if not impossible.
Thanks,
Nicolai
--
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg), GF: Felix Imendörffer
On 3/2/20 6:12 PM, Petr Mladek wrote:
On Mon 2020-03-02 16:45:24, JeffleXu wrote:
quoted
Hello guys,
I'm new to livepatch world and now I'm a little confused with livepatch's
per-task consistency model which is introduced by [1]. I've also readed the
discussion on mailing list [2], which introduces shadow variable to handle
data layout and semantic changes. But there's still some confusion with this
per-task consistency model.
According to the model, there will be scenario where old function and new
function can co-exist, though for a single thread, it sees either all new
functions or all old functions.
I can't understand why Vojtech said that 'old func processing new data' was
impossible. Assuming a scenario where a process calls func-A to submit a
work request (inserted into a global list), and then a kthread is
responsible
for calling func-B to process all work requests in the list. What if this
process
has finished the transition (sees new func-A) while kthread still sees the
old func-B?
In this case, old func-B has to process new data. If there's some lock
semantic
changes in func-A and func-B, then old func-B has no way identifying the
shadow
variable labeled by new func-A.
Please tell me if I missed something, and any suggestions will be
appreciated. ;)
No, you did not miss anything. The consistency is only per-thread,
If a livepatch is changing semantic of a global variable it must
allow doing the changes only when the entire system is using
the new code. And the new code must be able to deal with both
old and new data.
The new data semantic can be enable by post-patch callback
that is called when the transition has finished.
Thanks for replying. I didn't consider callback earlier, and yes it
works in this case.
Thanks.
Jeffle
Thanks for replying. ;)
On 3/2/20 6:36 PM, Nicolai Stange wrote:
Hi,
JeffleXu [off-list ref] writes:
quoted
According to the model, there will be scenario where old function and new
function can co-exist, though for a single thread, it sees either all new
functions or all old functions.
That's correct.
quoted
I can't understand why Vojtech said that 'old func processing new
data' was impossible.
Just to make it explicit: Vojtech was talking about data layout
changes.
Fine
That is, consider you have something like e.g. this in the unmodified
kernel sources:
struct my_driver_work
{
struct work_struct work;
struct list_head works_list;
void *some_payload;
};
In general, you can't change that struct definition from a live
patch. So simply extending it like this
struct my_driver_work
{
struct work_struct work;
struct list_head works_list;
unsigned long flags /* New */
void *some_payload;
};
won't work.
quoted
Assuming a scenario where a process calls func-A to submit a
work request (inserted into a global list), and then a kthread is
responsible
for calling func-B to process all work requests in the list. What if
this process
has finished the transition (sees new func-A) while kthread still sees
the old func-B?
Going with the example from above, the patched func-A would submit
instances of the new struct my_driver_work whereas the unpatched func-B
would expect the ->some_payload pointer at where ->flags is stored now,
which is bad, obviously.
In this specific example, it could perhaps be possible to make the
patched func-A associate a shadow variable corresponding to the new
->flags member with newly created struct my_driver_work instances (of
original, unmodified layout). Any unpatched func-B would obviously
ignore it and consider the shadow only when it becomes patched. It very
much depends on the specific situation whether or not this is
acceptable. If not, the ->post_patch() can sometimes be used to achieve
some notion of a "global consistency" state (in this context,
c.f. Documentation/livepatch/system-state.rst).
Well, I'm familiar with shadow variable, but didn't consider callbacks
earlier. Since the version of my
kernel is not new enough, the "system state API" has not been merged in
my kernel. I will read it later.
Note however, that the patched func-B must always be able to handle
the situation where a struct my_driver_work instance does not have such
a ->flags shadow attached to it, either because the instance had been
created when the live patch has not been applied at all or because it
has been submitted from a not yet transitioned func-A.
There's another subtlety: the deallocation code for struct
my_driver_work needs to get livepatched as well to make it free the
->flags shadow. Consider the case where func-A has been transitioned,
but the deallocation code hasn't yet. Without any extra measures in
func-A, it could happen that a stale ->flags shadow from a deallocated
struct my_driver_work gets (wrongly) reassociated with a fresh struct
my_driver_work instance allocated at the same address as the old one
(because shadow variables are keyed on addresses of the data they're
associated with). Sometimes that's acceptable, sometimes it's not. In
the latter case you probably had to check for this situation and work
around it in the allocation code, i.e. the live-patched func-A.
I've never thought about this. It's a valuable suggestion.
Finally, let me remark that from my experience, most CVEs (>95%) can be
fixed via live patching without having to resort to either of shadow
variables, callbacks or the state API. For the rest, things usually tend
to become really non-trivial, hackish and subtle.
Thanks for your experience.
quoted
In this case, old func-B has to process new data. If there's some lock
semantic
changes in func-A and func-B, then old func-B has no way identifying
the shadow
variable labeled by new func-A.
I don't understand what you mean by "variable labeled by new func-A"?
Anyway, it's correct that an unpatched func-B would not consider any
shadow variables instantiated by patched func-A. And it's also correct
that changing locking semantics is difficult, if not impossible.
Just means shadow variable allocated by new patched function. I know
shadow variable can serve
as a flag to enable the new functions, but in this case the post_patch()
callback is obviously more
appropriate to serve as this role.
So as far as I understand, for all kinds of (data/locking) semantic
changes, it's the responsibility of the
patch writer to detect the semantic changes, and usually it can only be
analyzed case by case. Right?
JeffleXu [off-list ref] writes:
So as far as I understand, for all kinds of (data/locking) semantic
changes, it's the responsibility of the patch writer to detect the
semantic changes, and usually it can only be analyzed case by
case. Right?
Exactly :)
Nicolai
--
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg), GF: Felix Imendörffer