From: Andrew Morton <hidden> Date: 2015-07-08 00:07:46
On Wed, 08 Jul 2015 09:02:59 +0900 Gioh Kim [off-list ref] wrote:
2015-07-08 ______ 7:37___ Andrew Morton ___(___) ___ ___:
quoted
On Tue, 7 Jul 2015 13:36:20 +0900 Gioh Kim [off-list ref] wrote:
quoted
From: Gioh Kim <redacted>
Hello,
This series try to enable migration of non-LRU pages, such as driver's page.
My ARM-based platform occured severe fragmentation problem after long-term
(several days) test. Sometimes even order-3 page allocation failed. It has
memory size 512MB ~ 1024MB. 30% ~ 40% memory is consumed for graphic processing
and 20~30 memory is reserved for zram.
I found that many pages of GPU driver and zram are non-movable pages. So I
reported Minchan Kim, the maintainer of zram, and he made the internal
compaction logic of zram. And I made the internal compaction of GPU driver.
They reduced some fragmentation but they are not enough effective.
They are activated by its own interface, /sys, so they are not cooperative
with kernel compaction. If there is too much fragmentation and kernel starts
to compaction, zram and GPU driver cannot work with the kernel compaction.
...
This patch set is tested:
- turn on Ubuntu 14.04 with 1G memory on qemu.
- do kernel building
- after several seconds check more than 512MB is used with free command
- command "balloon 512" in qemu monitor
- check hundreds MB of pages are migrated
OK, but what happens if the balloon driver is not used to force
compaction? Does your test machine successfully compact pages on
demand, so those order-3 allocations now succeed?
If any driver that has many pages like the balloon driver is forced to compact,
the system can get free high-order pages.
I have to show how this patch work with a driver existing in the kernel source,
for kernel developers' undestanding. So I selected the balloon driver
because it has already compaction and working with kernel compaction.
I can show how driver pages is compacted with lru-pages together.
Actually balloon driver is not best example to show how this patch compacts pages.
The balloon driver compaction is decreasing page consumtion, for instance 1024MB -> 512MB.
I think it is not compaction precisely. It frees pages.
Of course there will be many high-order pages after 512MB is freed.
Can the various in-kernel GPU drivers benefit from this? If so, wiring
up one or more of those would be helpful?
On Wed, 08 Jul 2015 09:02:59 +0900 Gioh Kim [off-list ref] wrote:
quoted
2015-07-08 ______ 7:37___ Andrew Morton ___(___) ___ ___:
quoted
On Tue, 7 Jul 2015 13:36:20 +0900 Gioh Kim [off-list ref] wrote:
quoted
From: Gioh Kim <redacted>
Hello,
This series try to enable migration of non-LRU pages, such as driver's page.
My ARM-based platform occured severe fragmentation problem after long-term
(several days) test. Sometimes even order-3 page allocation failed. It has
memory size 512MB ~ 1024MB. 30% ~ 40% memory is consumed for graphic processing
and 20~30 memory is reserved for zram.
I found that many pages of GPU driver and zram are non-movable pages. So I
reported Minchan Kim, the maintainer of zram, and he made the internal
compaction logic of zram. And I made the internal compaction of GPU driver.
They reduced some fragmentation but they are not enough effective.
They are activated by its own interface, /sys, so they are not cooperative
with kernel compaction. If there is too much fragmentation and kernel starts
to compaction, zram and GPU driver cannot work with the kernel compaction.
...
This patch set is tested:
- turn on Ubuntu 14.04 with 1G memory on qemu.
- do kernel building
- after several seconds check more than 512MB is used with free command
- command "balloon 512" in qemu monitor
- check hundreds MB of pages are migrated
OK, but what happens if the balloon driver is not used to force
compaction? Does your test machine successfully compact pages on
demand, so those order-3 allocations now succeed?
If any driver that has many pages like the balloon driver is forced to compact,
the system can get free high-order pages.
I have to show how this patch work with a driver existing in the kernel source,
for kernel developers' undestanding. So I selected the balloon driver
because it has already compaction and working with kernel compaction.
I can show how driver pages is compacted with lru-pages together.
Actually balloon driver is not best example to show how this patch compacts pages.
The balloon driver compaction is decreasing page consumtion, for instance 1024MB -> 512MB.
I think it is not compaction precisely. It frees pages.
Of course there will be many high-order pages after 512MB is freed.
Can the various in-kernel GPU drivers benefit from this? If so, wiring
up one or more of those would be helpful?
I'm sure that other in-kernel GPU drivers can have benefit.
It must be helpful.
If I was familiar with other in-kernel GPU drivers code, I tried to patch them.
It's too bad.
Minchan Kim said he had a plan to apply this patch into zram compaction.
Many embedded machines use several hundreds MB for zram.
The zram can also have benefit with this patch as much as GPU drivers.
From: Minchan Kim <minchan@kernel.org> Date: 2015-07-08 00:35:08
On Wed, Jul 08, 2015 at 09:19:50AM +0900, Gioh Kim wrote:
2015-07-08 오전 9:07에 Andrew Morton 이(가) 쓴 글:
quoted
On Wed, 08 Jul 2015 09:02:59 +0900 Gioh Kim [off-list ref] wrote:
quoted
2015-07-08 ______ 7:37___ Andrew Morton ___(___) ___ ___:
quoted
On Tue, 7 Jul 2015 13:36:20 +0900 Gioh Kim [off-list ref] wrote:
quoted
From: Gioh Kim <redacted>
Hello,
This series try to enable migration of non-LRU pages, such as driver's page.
My ARM-based platform occured severe fragmentation problem after long-term
(several days) test. Sometimes even order-3 page allocation failed. It has
memory size 512MB ~ 1024MB. 30% ~ 40% memory is consumed for graphic processing
and 20~30 memory is reserved for zram.
I found that many pages of GPU driver and zram are non-movable pages. So I
reported Minchan Kim, the maintainer of zram, and he made the internal
compaction logic of zram. And I made the internal compaction of GPU driver.
They reduced some fragmentation but they are not enough effective.
They are activated by its own interface, /sys, so they are not cooperative
with kernel compaction. If there is too much fragmentation and kernel starts
to compaction, zram and GPU driver cannot work with the kernel compaction.
...
This patch set is tested:
- turn on Ubuntu 14.04 with 1G memory on qemu.
- do kernel building
- after several seconds check more than 512MB is used with free command
- command "balloon 512" in qemu monitor
- check hundreds MB of pages are migrated
OK, but what happens if the balloon driver is not used to force
compaction? Does your test machine successfully compact pages on
demand, so those order-3 allocations now succeed?
If any driver that has many pages like the balloon driver is forced to compact,
the system can get free high-order pages.
I have to show how this patch work with a driver existing in the kernel source,
for kernel developers' undestanding. So I selected the balloon driver
because it has already compaction and working with kernel compaction.
I can show how driver pages is compacted with lru-pages together.
Actually balloon driver is not best example to show how this patch compacts pages.
The balloon driver compaction is decreasing page consumtion, for instance 1024MB -> 512MB.
I think it is not compaction precisely. It frees pages.
Of course there will be many high-order pages after 512MB is freed.
Can the various in-kernel GPU drivers benefit from this? If so, wiring
up one or more of those would be helpful?
I'm sure that other in-kernel GPU drivers can have benefit.
It must be helpful.
If I was familiar with other in-kernel GPU drivers code, I tried to patch them.
It's too bad.
Minchan Kim said he had a plan to apply this patch into zram compaction.
Many embedded machines use several hundreds MB for zram.
The zram can also have benefit with this patch as much as GPU drivers.
Hello Gioh,
It would be helpful for fork-latency and zra+CMA in small memory system.
I will implement zsmalloc.migratepages after I finish current going works.
Thanks for the nice work!
--
Kind regards,
Minchan Kim
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
From: Dave Airlie <airlied@gmail.com> Date: 2015-07-08 22:48:00
quoted
Can the various in-kernel GPU drivers benefit from this? If so, wiring
up one or more of those would be helpful?
I'm sure that other in-kernel GPU drivers can have benefit.
It must be helpful.
If I was familiar with other in-kernel GPU drivers code, I tried to patch
them.
It's too bad.
I'll bring dri-devel into the loop here.
ARM GPU developers please take a look at this stuff, Laurent, Rob,
Eric I suppose.
Daniel Vetter you might have some opinions as well.
Dave.
Can the various in-kernel GPU drivers benefit from this? If so, wiring
up one or more of those would be helpful?
I'm sure that other in-kernel GPU drivers can have benefit.
It must be helpful.
If I was familiar with other in-kernel GPU drivers code, I tried to patch
them.
It's too bad.
I'll bring dri-devel into the loop here.
ARM GPU developers please take a look at this stuff, Laurent, Rob,
Eric I suppose.
I sent a patch, https://lkml.org/lkml/2015/3/24/1182, and my opinion about compaction
to ARM GPU developers via Korea ARM branch.
I got a reply that they had no time to review it.
I hope they're interested to this patch.
Daniel Vetter you might have some opinions as well.
Dave.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Daniel Vetter <hidden> Date: 2015-07-09 13:08:48
On Thu, Jul 09, 2015 at 08:55:25AM +0900, Gioh Kim wrote:
2015-07-09 오전 7:47에 Dave Airlie 이(가) 쓴 글:
quoted
quoted
quoted
Can the various in-kernel GPU drivers benefit from this? If so, wiring
up one or more of those would be helpful?
I'm sure that other in-kernel GPU drivers can have benefit.
It must be helpful.
If I was familiar with other in-kernel GPU drivers code, I tried to patch
them.
It's too bad.
I'll bring dri-devel into the loop here.
ARM GPU developers please take a look at this stuff, Laurent, Rob,
Eric I suppose.
I sent a patch, https://lkml.org/lkml/2015/3/24/1182, and my opinion about compaction
to ARM GPU developers via Korea ARM branch.
I got a reply that they had no time to review it.
I hope they're interested to this patch.
i915 gpus would support 64kb and 2mb pages, but we never implemented this.
I don't think this would fit for gem based drivers since our backing
storage is shmemfs. So if we want to implement page migration (which we'd
probably want to make large pages work well) we'd need to pimp shmem to a)
hand large pages to us b) forward the migrate calls. Probably that means
we need to build our own gemfs reusing shmemfs code.
I guess something similar would apply for ttm-based drivers (which use
shmemfs just for swap-in/out but otherwise have their own page allocator,
at least sometimes).
Given all that I'd expect anything implementing migrate to just create a
gpufs thing for the backing storage, no need for more hooks. There's also
other areas for better code sharing among gpu drivers (e.g. mmu notifiers
to get off userspace pages slurped in&pinned with gup or shrinker
callbacks to get the gpu off it's memory binge). But that would all be
helper libraries in drm, not sure we need anything new from the core vm.
Also there's a bit a lack of gpu drivers from the arm world in upstream,
which is probabyl why this patch series doesn't come with a user. Might be
better to first upstream the driver before talking about additional
infrastructure that it needs.
-Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Ville Syrjälä <hidden> Date: 2015-07-09 13:33:01
On Thu, Jul 09, 2015 at 03:08:48PM +0200, Daniel Vetter wrote:
On Thu, Jul 09, 2015 at 08:55:25AM +0900, Gioh Kim wrote:
quoted
2015-07-09 오전 7:47에 Dave Airlie 이(가) 쓴 글:
quoted
quoted
quoted
Can the various in-kernel GPU drivers benefit from this? If so, wiring
up one or more of those would be helpful?
I'm sure that other in-kernel GPU drivers can have benefit.
It must be helpful.
If I was familiar with other in-kernel GPU drivers code, I tried to patch
them.
It's too bad.
I'll bring dri-devel into the loop here.
ARM GPU developers please take a look at this stuff, Laurent, Rob,
Eric I suppose.
I sent a patch, https://lkml.org/lkml/2015/3/24/1182, and my opinion about compaction
to ARM GPU developers via Korea ARM branch.
I got a reply that they had no time to review it.
I hope they're interested to this patch.
i915 gpus would support 64kb and 2mb pages, but we never implemented this.
I don't think this would fit for gem based drivers since our backing
storage is shmemfs. So if we want to implement page migration (which we'd
probably want to make large pages work well) we'd need to pimp shmem to a)
hand large pages to us b) forward the migrate calls. Probably that means
we need to build our own gemfs reusing shmemfs code.
AFAIK there are efforts ongoing to make large pages work with shmem.
Kirill, IIRC you mentioned that you're were looking into this a while
back?
--
Ville Syrjälä
Intel OTC
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Kirill A. Shutemov <hidden> Date: 2015-07-09 14:02:48
On Thu, Jul 09, 2015 at 04:33:01PM +0300, Ville Syrjälä wrote:
On Thu, Jul 09, 2015 at 03:08:48PM +0200, Daniel Vetter wrote:
quoted
On Thu, Jul 09, 2015 at 08:55:25AM +0900, Gioh Kim wrote:
quoted
2015-07-09 오전 7:47에 Dave Airlie 이(가) 쓴 글:
quoted
quoted
quoted
Can the various in-kernel GPU drivers benefit from this? If so, wiring
up one or more of those would be helpful?
I'm sure that other in-kernel GPU drivers can have benefit.
It must be helpful.
If I was familiar with other in-kernel GPU drivers code, I tried to patch
them.
It's too bad.
I'll bring dri-devel into the loop here.
ARM GPU developers please take a look at this stuff, Laurent, Rob,
Eric I suppose.
I sent a patch, https://lkml.org/lkml/2015/3/24/1182, and my opinion about compaction
to ARM GPU developers via Korea ARM branch.
I got a reply that they had no time to review it.
I hope they're interested to this patch.
i915 gpus would support 64kb and 2mb pages, but we never implemented this.
I don't think this would fit for gem based drivers since our backing
storage is shmemfs. So if we want to implement page migration (which we'd
probably want to make large pages work well) we'd need to pimp shmem to a)
hand large pages to us b) forward the migrate calls. Probably that means
we need to build our own gemfs reusing shmemfs code.
AFAIK there are efforts ongoing to make large pages work with shmem.
Kirill, IIRC you mentioned that you're were looking into this a while
back?
I work in this direction, but don't have anything to show at the moment.
Hugh has published his implementation of huge tmpfs back in February:
http://lkml.kernel.org/g/alpine.LSU.2.11.1502201941340.14414@eggly.anvils
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
Also there's a bit a lack of gpu drivers from the arm world in upstream,
which is probabyl why this patch series doesn't come with a user. Might be
better to first upstream the driver before talking about additional
infrastructure that it needs.
-Daniel
I'm not from ARM but I just got the idea of driver page migration
during I worked with ARM gpu driver.
I'm sure this patch is good for zram and balloon
and hope it can be applied to drivers consuming many pages and generating fragmentation,
such as GPU or gfx driver.
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html