[RFC] Virtual CRTCs (proposal + experimental code)

Hi everyone,

I would like to bring to the attention of dri-devel and linux-fbdev
community a set of hopefully useful and interesting patches that
I (and a few other colleagues) have been working on during the past
few months. Here, I will provide a short abstract, so that you can
decide whether this is of interest for you. At the end, I will
provide the pointers to the code and documentation.

The code is based on Dave Arilie's tree, drm-next branch and it
allows a GPU driver to have an arbitrary number of CRTCs
(configurable by user) instead of only those CRTCs that represent
real hardware.

The new CRTCs, that we call Virtual CRTCs can be attached to a
foreign device, that we call CTD device (short for Compression
Transmission and Display), and pixels can be streamed out of the GPU
to the device.

In one example, we use AMD/ATI Radeon GPU to do 3D rendering
(accelerated, of course) and we use our code to add additional
monitor heads using DisplayLink devices. In other words, we achieve
accelerated 3D rendering on a DisplayLink monitor. In another example
we funnel rendered pixels to userland by emulating a Video-for-Linux
device (and then userland can do whatever it wants with it). While
doing all this, GPU has no idea that we are doing this, the entire DRI
"thinks" that it is just dealing with a GPU that has a few "extra"
connectors and CRTCs. So everything ports without the need to modify
anything in the userland.

In general any device that can do something good with rendered pixels
can act as a CTD device, allowing a GPU to be an acceleration device
for a less capable display device or (the opposite) a frame-buffer-based
display device to be an expansion card for a GPU. Of course, for
each display device, a driver would have to be made compatible with our
new infrastructure (which is what we have done with DisplayLink driver
and also wrote one "synthetic" driver to fake out a V4L2 device as a
CTD device).

The newly introduced kernel module that we call VCRTCM (short for

42 more lines

Hi,

Quick question: How does this compare to Dave Airlies' PRIME buffer
sharing work for drm respectively the more generic dma_buf buffer sharing
work pushed by Linaro? You seem to aim for a solution for similar problems
(judging by your description) using a rather different approach.

Cheers, Daniel

On Thu, Nov 03, 2011 at 11:59:21AM -0400, Ilija Hadzic wrote:

Hi everyone,

I would like to bring to the attention of dri-devel and linux-fbdev
community a set of hopefully useful and interesting patches that
I (and a few other colleagues) have been working on during the past
few months. Here, I will provide a short abstract, so that you can
decide whether this is of interest for you. At the end, I will
provide the pointers to the code and documentation.

The code is based on Dave Arilie's tree, drm-next branch and it
allows a GPU driver to have an arbitrary number of CRTCs
(configurable by user) instead of only those CRTCs that represent
real hardware.

The new CRTCs, that we call Virtual CRTCs can be attached to a
foreign device, that we call CTD device (short for Compression
Transmission and Display), and pixels can be streamed out of the GPU
to the device.

In one example, we use AMD/ATI Radeon GPU to do 3D rendering
(accelerated, of course) and we use our code to add additional
monitor heads using DisplayLink devices. In other words, we achieve
accelerated 3D rendering on a DisplayLink monitor. In another example
we funnel rendered pixels to userland by emulating a Video-for-Linux
device (and then userland can do whatever it wants with it). While
doing all this, GPU has no idea that we are doing this, the entire DRI
"thinks" that it is just dealing with a GPU that has a few "extra"
connectors and CRTCs. So everything ports without the need to modify
anything in the userland.

In general any device that can do something good with rendered pixels
can act as a CTD device, allowing a GPU to be an acceleration device
for a less capable display device or (the opposite) a frame-buffer-based
display device to be an expansion card for a GPU. Of course, for
each display device, a driver would have to be made compatible with our
new infrastructure (which is what we have done with DisplayLink driver
and also wrote one "synthetic" driver to fake out a V4L2 device as a
CTD device).

The newly introduced kernel module that we call VCRTCM (short for

46 more quoted lines

-- 
Daniel Vetter
Mail: daniel@ffwll.ch
Mobile: +41 (0)79 365 57 48

Hi everyone,

I would like to bring to the attention of dri-devel and linux-fbdev
community a set of hopefully useful and interesting patches that
I (and a few other colleagues) have been working on during the past
few months. Here, I will provide a short abstract, so that you can
decide whether this is of interest for you. At the end, I will
provide the pointers to the code and documentation.

The code is based on Dave Arilie's tree, drm-next branch and it
allows a GPU driver to have an arbitrary number of CRTCs
(configurable by user) instead of only those CRTCs that represent
real hardware.

Well the current plan I had for this was to do it in userspace, I don't think the kernel
has any business doing it and I think for the simple USB case its fine but will fallover
when you get to the non-trivial cases where some sort of acceleration is required to move
pixels around. But in saying that its good you've done what something, and I'll try and spend
some time reviewing it.

The current plan from my POV is to add hotplug support to the X server and just hotplug USB
devices up there, and not mess up the kernel with lots of extra state that really the drivers
don't need to know about.

I'm also not sure how you deal with tiling etc, you can also start hitting rendering limits,
where a GPU can render to 4kx4k but you can plug in more USB devices, again I'm hoping to
solve this in userspace as well.

But I'll take some time if I can find it to look over it.

Dave.

Well the current plan I had for this was to do it in userspace, I don't think the kernel
has any business doing it and I think for the simple USB case its fine but will fallover
when you get to the non-trivial cases where some sort of acceleration is required to move
pixels around. But in saying that its good you've done what something, and I'll try and spend
some time reviewing it.

There are some clear advantages in the kernel doing bits of this I think.
The kernel understands device to device DMA, and has a better idea than
userspace about things like buffer alignment internals. It also means
this ultimately can work without X running which is a plus for some
applications (I want a displaylink gadget for my phone but thats another
story 8)).

I'm also not sure how you deal with tiling etc, you can also start hitting rendering limits,
where a GPU can render to 4kx4k but you can plug in more USB devices, again I'm hoping to
solve this in userspace as well.

Tiling has to be handled by the recipient (at least when the fb is shared). The nastier end of
it that I don't see convered in the documentation is the handling of
fencing between cards. Eg if you wanted to do display on one card fed
into a second to do effects processing (think about TV type stuff)

Alan

On Thu, 3 Nov 2011, David Airlie wrote:

Well the current plan I had for this was to do it in userspace, I don't think the kernel
has any business doing it and I think for the simple USB case its fine but will fallover
when you get to the non-trivial cases where some sort of acceleration is required to move
pixels around. But in saying that its good you've done what something, and I'll try and spend
some time reviewing it.

The reason I opted for doing this in kernel is that I wanted to confine 
all the changes to a relatively small set of modules. At first this was a 
pragmatic approach, because I live out of the mainstream development tree 
and I didn't want to turn my life into an ethernal 
merging/conflict-resolution activity.

However, a more fundamental reason for it is that I didn't want to be tied 
to X. I deal with some userland applications (that unfortunately I can't 
provide much detail of .... yet) that live directly on the top of libdrm.

So I set myself a goal of "full application transparency". Whatever is 
thrown at me, I wanted to be able to handle without having to touch any 
piece of application or library that the application relies on.

I think I have achieved this goal and really everything I tried just 
worked out of the box (with an exception of two bug fixes to ATI DDX
and Xorg, that are bugs with or without my work).

-- Ilija

In general any device that can do something good with rendered pixels
can act as a CTD device, allowing a GPU to be an acceleration device
for a less capable display device or (the opposite) a frame-buffer-based
display device to be an expansion card for a GPU. Of course, for
each display device, a driver would have to be made compatible with our
new infrastructure (which is what we have done with DisplayLink driver
and also wrote one "synthetic" driver to fake out a V4L2 device as a
CTD device).

So I expect the way I'd like this to work, is that we have full blown drm
drivers for all the devices and then some sort of linkage layer between them,
so one driver can export crtcs from another, instead of having special case
ctd drivers. Now I can see the reason for the v4l one, but I have for example
a udl kms driver, and I'd like to have it work alongside this stuff,
and userspace
could bind the crtcs from it to another driver. I'm not sure how much work
this would be or if its just a matter of adding a CTD interface to the udl kms
device.

Dave

On Mon, 7 Nov 2011, Dave Airlie wrote:

So I expect the way I'd like this to work, is that we have full blown drm
drivers for all the devices and then some sort of linkage layer between them,
so one driver can export crtcs from another, instead of having special case
ctd drivers.

I agree and that is actually a long-term plan from our side. CTD 
functionality should be integral part of existing drivers not the new 
driver, unless there is a new functionality that makes sense as CTD-only 
(like v4l2ctd).

In the world that I imagine, likage layer is VCRTCM. Unaccelerated 
frabebuffer devices (UDL for example, but in general anything that "lives" 
in fbdev world) can choose (based on some policy from userland) whether to 
act as CTD driver and register themselves with VCRTCM (when they want 
acceleration "assistance" from a GPU in the system) or to load as
normal fbdev devices (when they want to run on their own).

Now I can see the reason for the v4l one, but I have for example
a udl kms driver, and I'd like to have it work alongside this stuff,
and userspace
could bind the crtcs from it to another driver. I'm not sure how much work
this would be or if its just a matter of adding a CTD interface to the udl kms
device.

The only reason we wrote a new udlctd driver was  because it was quicker 
that way (we just ripped some code from udlfb driver and added our CTD 
functionality).

The plan was always to merge udlctd and udlfb at some point, but first 
I'd like to see how perceptive is the community to the concept. If the 
concept makes sense, then we'll throw in enough programming to consolidate 
the drivers. Nobody wants three competing drivers for the same device 
(udlfb, your udl kms driver and our udlctd).

Speaking of udl driver, is your udl-v2 branch competing with udlfb? 
Externally, they seem to do similar or the same thing, but one is based on 
DRM (and I guess the required DDX is xf86-video-modesetting), while the 
other is based on fbdev and the required DDX is xf8b-video-fbdev. While 
from my perspective both could be consilidated with a CTD functionality, 
but it makes me wonder in which direction is the community development 
moving: is everything from fbdev-world moving under DRM as a set of 
unaccelerated KMS drivers or is fbdev staying separate for good ?
Depending on what the trend is, one or the other udl driver (udl-v2 or 
udlfb) will make more sense to be merged with udlctd.

-- Ilija

I would like to bring to the attention of dri-devel and linux-fbdev
community a set of hopefully useful and interesting patches that
I (and a few other colleagues) have been working on during the past
few months. Here, I will provide a short abstract, so that you can
decide whether this is of interest for you. At the end, I will
provide the pointers to the code and documentation.

The code is based on Dave Arilie's tree, drm-next branch and it
allows a GPU driver to have an arbitrary number of CRTCs
(configurable by user) instead of only those CRTCs that represent
real hardware.

So another question I have is how you would intend this to work from a
user POV, like how it would integrate with a desktop environment, X or
wayland, i.e. with little or no configuration.

I still foresee problems with tiling, we generally don't encourage
accel code to live in the kernel, and you'll really want a
tiled->untiled blit for this thing, also for Intel GPUs where you have
UMA, would you read from the UMA.

It also doesn't solve the optimus GPU problem in any useful fashion,
since it can't deal with all the use cases, so we still have to write
an alternate solution that can deal with them, so we just end up with
two answers.

Dave.

On Wed, 23 Nov 2011, Dave Airlie wrote:

So another question I have is how you would intend this to work from a
user POV, like how it would integrate with a desktop environment, X or
wayland, i.e. with little or no configuration.

First thing to understand is that when a virtual CRTC is created, it looks 
to the user like the GPU has an additional DisplayPort connector.
At present I "abuse" DisplayPort, but I have seen that you pushed a patch 
from VMware that adds Virtual connector, so eventually I'll switch to that 
naming. The number of virtual CRTCs is determined when the driver loads 
and that is a static configuration parameter. This does not restrict the 
user because unused virutal CRTCs are just like disconnected connectors on 
the GPU. In extreme case, a user could max out the number of virtual CRTCs 
(i.e. 32 minus #-of-physical-CRTCs), but in general the system needs to be 
booted with maximum number of anticipated CRTCs. Run-time addition and 
removal of CRTCs is not supported at this time and that would be much 
harder to implement and would affect the whole DRM module everywhere.

So now we have a system that booted up and DRM sees all of its real 
connectors as well as virtual ones (as DisplayPorts at present). If there 
is no CTD device attached to virtual CRTCs, these virtual connectors are 
disconnected as far as DRM is concerned. Now the userspace must call 
"attach/fps" ioctl to associate CTDs with CRTCs. I'll explain shortly how 
to automate that and how to eliminate the burden from the user, but for 
now, please assume that "attach/fps" gets called from userland somehow.

When the attach happens, that is a hotplug event (VCRTCM generates it) to 
DRM, just like someone plugged in the monitor. Then when XOrg starts, it
will use the DisplayPort that represents a virtual CRTC just like any 
other connector. How it will use it, will depend on what the xorg.conf 
says, but the key point is that this connector is no different from any 
other connector that the GPU provides and is thus used as an "equal 
citizen". No special configuration is necessary once attached to CTD.

If CTD is detached and new CTD attached, that is just like yanking out 
monitor cable and plugging in the new one. DRM will get all hotplug events 
and windowing system will do the same thing it would normally do with any 
other port. If RANDR is called to resize the desktop it will also work and 
X will have no idea that one of the connectors is on a virtual CRTC. I 
also have another feature, where when CTD is attached, it can ask the 
device it drives for the connection status and propagate that all the way 
back to DRM (this is useful for CTD devices that drive real monitors, like 
DisplayLink).

46 more lines

I still foresee problems with tiling, we generally don't encourage
accel code to live in the kernel, and you'll really want a
tiled->untiled blit for this thing,

Accel code should not go into the kernel (that I fully agree) and there is 
nothing here that would behove us to do so. Restricting my comments to 
Radeon GPU (which is the only one that I know well enough), shaders for 
blit copy live in the kernel and irrespective of VCRTCM work. I rely on 
them to move the frame buffer out of VRAM to CTD device but I don't add 
any additional features.

Now for detiling, I think that it should be the responsibility of the 
receiving CTD device, not the GPU pushing the data (Alan mentioned that 
during the initial set of comments, and although I didn't say anything to 
it that has been my view as well).

Even if you wanted to use GPU for detiling (which I'll explain shortly why 
you should not), it would not require any new accel code in the kernel. It 
would merely require one bit flip in the setup of blit copy that already 
lives in the kernel.

However, de-tiling in GPU is a bad idea for two reasons. I tried to do 
that just as an experiment on Radeon GPUs and watched with the PCI Express 
analyzer what happens on the bus (yeah, I have some "heavy weapons" in my 
lab). Normally a tile is a continuous array of memory locations in VRAM. 
If blit-copy function is told to assume tiled source and linear 
destination (de-tiling) it will read a continuous set of addresses in 
VRAM, but then scatter 8 rows of 8 pixels each on non-contignuous set of 
addresses of the destination. If the destination is the PCI-Express bus, 
it will result in 8 32-byte write transactions instead of 2 128-byte 
transactions per each tile. That will choke the throughput of the bus 
right there.

BTW, this is the crux of the blit-copy performance improvement that you 
got from me back in October. Since blit-copy deals with copying a linear 
array, playing with tiled/non-tiled bits only affects the order in which 
addresses are accessed, so the trick was to get rid of short PCIe 
transactions and also shape up linear to rectangle mapping to make address 
pattern more friendly for the host.

also for Intel GPUs where you have
UMA, would you read from the UMA.

Yes the read would be from UMA. I have not yet looked at Intel GPUs in 
detail, so I don't have an answer for you on what problems would pop up 
and how to solve them, but I'll be glad to revisit the Intel discussion 
once I do some homework.

Some initial thoughts is that frame buffer in Intel are at the end of the 
day pages in the system memory, so anyone/anything can get to them if they 
are correctly mapped.

It also doesn't solve the optimus GPU problem in any useful fashion,
since it can't deal with all the use cases, so we still have to write
an alternate solution that can deal with them, so we just end up with
two answers.

Can you elaborate on some specific use cases that are of your concern? I 
have had this case in mind and I think I can make it work. First I would 
have to add CTD functionality to Intel driver. That should be 
straightforward. Once I get there, I'll be ready to experiment and we'll 
probably be in better position to discuss the specifics then (i.e. when we 
have something working to compare with what you did in PRIME experiemnt), 
but it would be good to know your specific concerns early.


thanks,

Ilija

On Thu, Nov 24, 2011 at 5:59 AM, Ilija Hadzic
[off-list ref] wrote:

On Wed, 23 Nov 2011, Dave Airlie wrote:

quoted

So another question I have is how you would intend this to work from a
user POV, like how it would integrate with a desktop environment, X or
wayland, i.e. with little or no configuration.

First thing to understand is that when a virtual CRTC is created, it looks
to the user like the GPU has an additional DisplayPort connector.
At present I "abuse" DisplayPort, but I have seen that you pushed a patch
from VMware that adds Virtual connector, so eventually I'll switch to that
naming. The number of virtual CRTCs is determined when the driver loads and
that is a static configuration parameter. This does not restrict the user
because unused virutal CRTCs are just like disconnected connectors on the
GPU. In extreme case, a user could max out the number of virtual CRTCs (i.e.
32 minus #-of-physical-CRTCs), but in general the system needs to be booted
with maximum number of anticipated CRTCs. Run-time addition and removal of
CRTCs is not supported at this time and that would be much harder to
implement and would affect the whole DRM module everywhere.

So now we have a system that booted up and DRM sees all of its real
connectors as well as virtual ones (as DisplayPorts at present). If there is
no CTD device attached to virtual CRTCs, these virtual connectors are
disconnected as far as DRM is concerned. Now the userspace must call
"attach/fps" ioctl to associate CTDs with CRTCs. I'll explain shortly how to
automate that and how to eliminate the burden from the user, but for now,
please assume that "attach/fps" gets called from userland somehow.

When the attach happens, that is a hotplug event (VCRTCM generates it) to
DRM, just like someone plugged in the monitor. Then when XOrg starts, it
will use the DisplayPort that represents a virtual CRTC just like any other
connector. How it will use it, will depend on what the xorg.conf says, but
the key point is that this connector is no different from any other
connector that the GPU provides and is thus used as an "equal citizen". No
special configuration is necessary once attached to CTD.

If CTD is detached and new CTD attached, that is just like yanking out
monitor cable and plugging in the new one. DRM will get all hotplug events

6 more quoted lines

Okay so thats pretty much how I expected it to work, I don't think
Virtual makes sense for a displaylink attached device though,
again if you were using a real driver you would just re-use whatever
output type it uses, though I'm not sure how well that works,

Do you propogate full EDID information and all the modes or just the
supported modes? we use this in userspace to put monitor names in
GNOME display settings etc.

what does xrandr output looks like for a radeon GPU with 4 vcrtcs? do
you see 4 disconnected connectors? that again isn't a pretty user
experience.

So this is your hotplug demo, but the difference is that the new desktop can
use direct rendering. Also, everything that would work for a normal
connector works here without having to do any additional tricks. RANDR also
works seamlessly without having to do anything special. If you move away
from Xorg, to some other system (Wayland?), it still works for as long as
the new system knows how to deal with connectors that connect and
disconnect.

My main problem with this is as I'll explain below it only covers some
of the use cases, and I don't want a 50% solution at this point, by
doing something like this you are making it harder to get proper
support into something like wayland as they can ignore some of the
problems, however since this doesn't solve all the other problems it
means getting to a finished solution is actually less likely to
happen.

quoted

I still foresee problems with tiling, we generally don't encourage
accel code to live in the kernel, and you'll really want a
tiled->untiled blit for this thing,

Accel code should not go into the kernel (that I fully agree) and there is
nothing here that would behove us to do so. Restricting my comments to
Radeon GPU (which is the only one that I know well enough), shaders for blit
copy live in the kernel and irrespective of VCRTCM work. I rely on them to
move the frame buffer out of VRAM to CTD device but I don't add any
additional features.

Now for detiling, I think that it should be the responsibility of the
receiving CTD device, not the GPU pushing the data (Alan mentioned that
during the initial set of comments, and although I didn't say anything to it
that has been my view as well).

That is a pretty much a fundamental problem, there is no way you can
enumerate all the detiling necessary in the CTD device and there is no
way I'd want to merge that code to the kernel. r600 has 16 tiling
modes (we might only see 2 of these on scanout), r300->r500 have
another different set, r100->r200 have another different set, nouveau
has an major amount of modes, and intel has a full set and has crazy
memory configuration dependant swizzling, then gma500, etc. This just
won't be workable or scalable.

Even if you wanted to use GPU for detiling (which I'll explain shortly why
you should not), it would not require any new accel code in the kernel. It
would merely require one bit flip in the setup of blit copy that already
lives in the kernel.

That is fine for radeon, not so much for intel, nouveau etc.

However, de-tiling in GPU is a bad idea for two reasons. I tried to do that
just as an experiment on Radeon GPUs and watched with the PCI Express
analyzer what happens on the bus (yeah, I have some "heavy weapons" in my
lab). Normally a tile is a continuous array of memory locations in VRAM. If
blit-copy function is told to assume tiled source and linear destination
(de-tiling) it will read a continuous set of addresses in VRAM, but then
scatter 8 rows of 8 pixels each on non-contignuous set of addresses of the
destination. If the destination is the PCI-Express bus, it will result in 8
32-byte write transactions instead of 2 128-byte transactions per each tile.
That will choke the throughput of the bus right there.

The thing is this is how optimus works, the nvidia gpus have an engine
that you can program to move data from the nvidia tiled VRAM format to
the intel main memory tiled format, and make if efficent. radeon's
also have some engines that AMD so far haven't told us about, but
someone with no NDA with AMD could easily start REing that sort of
thing.

Yes the read would be from UMA. I have not yet looked at Intel GPUs in
detail, so I don't have an answer for you on what problems would pop up and
how to solve them, but I'll be glad to revisit the Intel discussion once I
do some homework.

Probably a good idea to do some more research on intel/nvidia GPUs.
With intel you can't read back from UMA since it'll be uncached memory
so unuseable, so you'll need to use the GPU to detile and move to some
sort of cached linear area you can readback from.

quoted

It also doesn't solve the optimus GPU problem in any useful fashion,
since it can't deal with all the use cases, so we still have to write
an alternate solution that can deal with them, so we just end up with
two answers.

Can you elaborate on some specific use cases that are of your concern? I
have had this case in mind and I think I can make it work. First I would
have to add CTD functionality to Intel driver. That should be
straightforward. Once I get there, I'll be ready to experiment and we'll
probably be in better position to discuss the specifics then (i.e. when we
have something working to compare with what you did in PRIME experiemnt),
but it would be good to know your specific concerns early.

Switchable/Optimus mode has two modes of operation,

a) nvidia GPU is rendering engine and the intel GPU is just used as a
scanout buffer for the LVDS panel. This mode is used when an external
digital display is plugged in, or in some plugged in configurations.

b) intel GPU is primary rendering engine, and the nvidia gpu is used
as an offload engine. This mode is used when on battery or power
saving, with no external displays plugged in. You can completely turn
on/off the nvidia GPU.

Moving between a and b has to be completely dynamic, userspace apps
need to deal with the whole world changing beneath them.

There is also switchable graphics mode, where there is a MUX used to
switch the outputs between the two GPUs.

So the main problem with taking all this code on-board is it sort of
solves (a), and (b) needs another bunch of work. Now I'd rather not
solve 50% of the issue and have future userspace apps just think they
can ignore the problem. As much as I dislike the whole dual-gpu setups
the fact is they exist and we can't change that, so writing userspace
to ignore the problem because its too hard isn't going to work. So if
I merge this VCRTC stuff I give a lot of people an excuse for not
bothering to fix the harder problems that hotplug and dynamic GPUs put
in front of you.

Dave.

thanks,

Ilija

On Thu, Nov 24, 2011 at 08:52:45AM +0000, Dave Airlie wrote:

So the main problem with taking all this code on-board is it sort of
solves (a), and (b) needs another bunch of work. Now I'd rather not
solve 50% of the issue and have future userspace apps just think they
can ignore the problem. As much as I dislike the whole dual-gpu setups
the fact is they exist and we can't change that, so writing userspace
to ignore the problem because its too hard isn't going to work. So if
I merge this VCRTC stuff I give a lot of people an excuse for not
bothering to fix the harder problems that hotplug and dynamic GPUs put
in front of you.

My 2 Rappen on this: I agree completely with your point that we should aim
for a full solution. GPU memory management across different devices is
hard, but solveable.

Furthermore I fear that a 50% solution that hides the memory management
and shuffling issues from userspace will end up being a leaky abstraction
(e.g. how and when is stuff transferred to the usb dp port, the kernel
might pin scanout buffers behind userspace's back screwing up the vram
accounting in userspace, random hotplugging of outputs ...). Also
v4l/embedded folks have similar issues (and the same tendency to just go
with a "simple" solution fitting their usecase) and with Intel dead-set on
entering the SoC market I'll have the joy to mess around with this stuff
pretty soon, too.

So I think we do have enough people interested in this and should be able
to cobble together something that does The Right Thing.

Cheers, Daniel
-- 
Daniel Vetter
Mail: daniel@ffwll.ch
Mobile: +41 (0)79 365 57 48

The thing is this is how optimus works, the nvidia gpus have an engine
that you can program to move data from the nvidia tiled VRAM format to

This is even more of a special case than DisplayLink ;-)

Probably a good idea to do some more research on intel/nvidia GPUs.
With intel you can't read back from UMA since it'll be uncached memory
so unuseable, so you'll need to use the GPU to detile and move to some
sort of cached linear area you can readback from.

It's main memory so there are various ways to read it or pull it into
cached space.

I merge this VCRTC stuff I give a lot of people an excuse for not
bothering to fix the harder problems that hotplug and dynamic GPUs put
in front of you.

I think both cases are slightly missing the mark, both are specialist
corner cases and once you add things like cameras to the mix that will
become even more painfully obvious.

The underlying need I think is a way to negotiate a shared buffer format
or pipeline between two devices. You also need in some cases to think
about shared fencing, and that is the bit that is really scary.

Figuring out the transform from A to B ('lets both use this buffer
format') or 'I can render then convert' is one thing. Dealing with two
GPUs firing into the same buffer while scanning it out I just pray
doesn't ever need shared fences.

Alan

On Thu, Nov 24, 2011 at 12:58 PM, Alan Cox [off-list ref] wrote:

quoted

The thing is this is how optimus works, the nvidia gpus have an engine
that you can program to move data from the nvidia tiled VRAM format to

This is even more of a special case than DisplayLink ;-)

quoted

Probably a good idea to do some more research on intel/nvidia GPUs.
With intel you can't read back from UMA since it'll be uncached memory
so unuseable, so you'll need to use the GPU to detile and move to some
sort of cached linear area you can readback from.

It's main memory so there are various ways to read it or pull it into
cached space.

We have no way to detile on the CPU for lots of intel corner cases, I don't hold
out for it being a proper solution, though in theory for hibernate its
a requirement to figure out.
But you can expose stuff via the GTT using fences to detile, but you
can't then get cached access to it.
So no really various ways, none of them useful or faster than getting
the GPU to blit somewhere linear
and flipping the dest mapping.

quoted

I merge this VCRTC stuff I give a lot of people an excuse for not
bothering to fix the harder problems that hotplug and dynamic GPUs put
in front of you.

I think both cases are slightly missing the mark, both are specialist
corner cases and once you add things like cameras to the mix that will
become even more painfully obvious.

The underlying need I think is a way to negotiate a shared buffer format
or pipeline between two devices. You also need in some cases to think
about shared fencing, and that is the bit that is really scary.

Figuring out the transform from A to B ('lets both use this buffer
format') or 'I can render then convert' is one thing. Dealing with two
GPUs firing into the same buffer while scanning it out I just pray
doesn't ever need shared fences.

But we have a project looking into all that, called dmabuf, we also
have the PRIME work which we hope to build on top of dmabuf.

The thing is there are lots of building blocks we need to put in
place, and we've mostly identified what they are, its just typing now.

Dave.

On Thu, 24 Nov 2011, Dave Airlie wrote:

Okay so thats pretty much how I expected it to work, I don't think
Virtual makes sense for a displaylink attached device though,
again if you were using a real driver you would just re-use whatever
output type it uses, though I'm not sure how well that works,

That is the consequence of the fact that virtual CRTCs are created at 
startup time when attached CTD is not known, while CTDs are attached at 
runtime. So when I register the virtual CRTC and the associated connector 
I have to use something for the connector type.

Admitting that my logic is biased by my design, to me "Virtual" connector 
type is an indicative that from GPU's perspective it's a connector that 
does not physically exist and is yet to be attached to some real display 
device. At that point the properties of the attached display become known 
to the system.

Do you propogate full EDID information and all the modes or just the
supported modes? we use this in userspace to put monitor names in
GNOME display settings etc.

Right now we propagate the entire list of modes that the attached CTD 
device has queried from the connected display (monitor). Propagating full 
EDID is really easy to add. That's if the CTD is driver for some real 
display. If CTD is just a "make-believe" display whose purpose is to be 
the conduit to some other pixel-processing component (e.g. V4L2CTD), then 
at some point in the chain we have to make up the set of modes that the 
logical display accepts and in that case the EDID does not exist by 
definition.

what does xrandr output looks like for a radeon GPU with 4 vcrtcs? do
you see 4 disconnected connectors? that again isn't a pretty user
experience.

Yes it shows 4 disconnected monitors. To me that is a logical consequence 
of the design in which virtual CRTCs and associated virtual connectors are 
always there. By know, it's clear to me that you are not too thrilled 
about it, but please allow me to turn the question back to you: in your 
solution with udl-v2 driver and a dedicated DDX for it, can you do the big 
desktop that spans across GPU's local and "foreign" displays and have 
acceleration on both? If not, what would it take you to get there and how 
complex the end result will be?

I'll get to Optimus/PRIME use case later, but if we for the moment focus 
on the use case in which a dumb framebuffer device extens the number of 
displays of a rendering-capable GPU, I think that VCRTCM offers quite a 
complete and universal solution and it is completely transparent with 
regard to the application, window manager, and display server.

Radeon + DisplayLink is the specific example. But in general it's Any GPU 
+ Any fbdev. It's not just one use case, it's a whole class of use cases
that would follow the same principle and for then the VCRTC alone 
suffices.

My main problem with this is as I'll explain below it only covers some
of the use cases, and I don't want a 50% solution at this point, by
doing something like this you are making it harder to get proper
support into something like wayland as they can ignore some of the
problems, however since this doesn't solve all the other problems it
means getting to a finished solution is actually less likely to
happen.

I presume that by 50% solution you are referring to Optimus/PRIME use 
case. That case actually consists of two related, but different problems. 
First is "render on node X and display on node Y" and the second is 
"dynamically and hitlessly switch rendering between node X and Y".

I have never claimed that VCRTCs solve the second problem (I could switch 
by restarting Xorg, but I know that this is not the solution you are 
looking for). I fully understand why you want both problems solved at the 
same time. However, I don't understand why solving one first would inhibit 
solving the other.

On the other hand, the Radeon + DisplayLink tandem use case (or in general 
GPU + fbdev tandem) consists only of "render on X, display on Y" problem. 
Here, you will probably say that there one can switch between hardware and 
software rendering and that it also has both problems. That is true, but 
unlike the Optimus/PRIME use case, using fbdev as a display extension to 
GPU is still useful alone. My point is that there is a value in solving 
first one problem and then follow with the other.

I think the crux of the problem is that you are not convinced that the
VCRTCM solution for problem #1 will make solving problem #2 easier and 
maybe you are afraid that it will make it harder. If that's a fair 
statement and if having me create an existence proof for problem #2 that 
still uses VCRTCM will help bring our positions closer, I am perfectly 
willing to do so .... I guess I've just signed up for some hacking ;-)

Note that for hitless GPU switching, I fully agree that support must be in 
userspace (you have to swap out paths in Mesa and DDX before even getting 
to kernel), but like I said, that is a separate problem from redirecting 
the display to another node.

r600 has 16 tiling
modes (we might only see 2 of these on scanout)

But VCRTC emulates a CRTC, so the only ones relevant are those that we see 
on the scnout. Do we really anticipate using all 16 for CRTC buffers ?

The thing is this is how optimus works, the nvidia gpus have an engine
that you can program to move data from the nvidia tiled VRAM format to
the intel main memory tiled format, and make if efficent. radeon's
also have some engines that AMD so far haven't told us about, but
someone with no NDA with AMD could easily start REing that sort of
thing.

If we could have every GPU efficiently push out pixels in some "common 
denominator" format that would be ideal, but at this time, the reality is 
far from it. Whether the obstacles are technical or legal, doesn't matter.

I fully understand your concern about the number of tiling/detiling 
combinations getting out of control, but I am not sure that the problem is 
as bad as you picture it if CRTC buffer uses only a subset of available 
tiling modes.

Switchable/Optimus mode has two modes of operation,

a) nvidia GPU is rendering engine and the intel GPU is just used as a
scanout buffer for the LVDS panel. This mode is used when an external
digital display is plugged in, or in some plugged in configurations.

b) intel GPU is primary rendering engine, and the nvidia gpu is used
as an offload engine. This mode is used when on battery or power
saving, with no external displays plugged in. You can completely turn
on/off the nvidia GPU.

Moving between a and b has to be completely dynamic, userspace apps
need to deal with the whole world changing beneath them.

So case a) is "render on X display on Y" problem and case b) (when NVidia 
is turned off, offload aside) is just a traditional rendering on one 
(Intel) GPU.  Real sticky point is dynamic switching and offload.

There is also switchable graphics mode, where there is a MUX used to
switch the outputs between the two GPUs.

One question for my education: I understand that MUX is essentially a 
switch outside the two GPUs that selects whether the output takes NVidia's 
"connector" or Intel's "connector", right ? When MUX is involved, then you 
don't have "render on X display on Y" problem at all, but it's only the 
"dynamic switching" problem. Is my understanding correct ?

There are also MUX-less laptops, where you only have case a)/b) that you 
described above, right ?

So the main problem with taking all this code on-board is it sort of
solves (a), and (b) needs another bunch of work. Now I'd rather not
solve 50% of the issue and have future userspace apps just think they
can ignore the problem. As much as I dislike the whole dual-gpu setups
the fact is they exist and we can't change that, so writing userspace
to ignore the problem because its too hard isn't going to work. So if
I merge this VCRTC stuff I give a lot of people an excuse for not
bothering to fix the harder problems that hotplug and dynamic GPUs put
in front of you.

Point taken. I still think that we are actually dealing with two separate 
problems, but you have your reasons why you want them solved together.
I also have a few more use cases which are solved with VCRTCM only and 
don't need dynamic switching, so these and also "3D Accel + DisplayLink" 
one will suffer by having to wait for full solution that covers all use 
cases, but that's your call and I don't question it.

I hope that you are not categorically dismissing an option that the 
solution can be implemented on the top of VCRTCM and that if I come back 
with some more code that shows that it addresses your concern that you 
will be perceptive to another round of review. I do appreciate you taking 
the time to look at this. I know that you are overbusy with day-to-day 
patches and merging.

I understand that the burden of building an existence proof falls on me 
and I am perfectly fine with that. BTW, If some poor soul reading this 
buys into my arguments and wants to join me in some hacking, I'd 
definitely welcome the collaboration ;-).

-- Ilija

So I think we do have enough people interested in this and should be able
to cobble together something that does The Right Thing.

We indeed have a non-trivial set of people interested in the same set of 
problems and each of us has partial and maybe competing solution. I want 
to make it clear that my, maybe disruptive and different from the 
plan-of-record, proposal should not be viewed as destructive or 
distracting. I am just offering to the community what I think is useful.

If this discussion sparks some joint effort that will bring us to the 
solution that everyone is happy with, even if no line of my code is found 
useful, I am perfectly fine with that (and I'll join the effort).

So at this point I think I should put out my back-of-the-napkin 
desiderata. That will hopefully shed some light on where I am coming from 
with VCRTCM proposal.

I want to be able to pull pixels out of the GPU and redirect them to an 
arbitrary device that can do something useful with them. This should not 
be limited to shooting photons into human eyeballs. I want to be able to 
run my applications without having to run X. I'd like the solution to be 
transparent to the application; that is, if I can write an application 
that can render something to a full screen, I want to redirect that 
"screen" to wherever I want without having to rewrite, recompile or relink 
the application. Actually, I want to do that redirection at runtime. I'd 
like to support all of the above in a way that it can also help solve more 
imminent shotcomings of Linux graphics system (Optimus, DisplayLink, etc. 
... cf. previous E-mails in this thread). I'd like it to work with 
multiple render nodes on the same GPU (something like Dave's multiseat 
work, in which both GPU and its display resources are virtual).

The logical consequence of this is that the render node and the display 
node should at some point become logically separate (different driver 
modules) even if they are physically on the same GPU. They are really two 
different subsystems that just happen to reside on the same circuit 
board, so it makes sense to separate them.

I don't think what I am saying is anything unique and what I said probably 
overlaps in good part with what others also want from the graphics 
subsystem. I can see the role of VCRTCM in all of the above, but I am 
open-minded. If we end up with a solution that has nothing to do with 
VCRTCM, I have no emotional ties with my code (and code of my colleagues 

5 more lines