On Mon, Aug 30, 2021 at 02:00:21PM +0200, Geert Uytterhoeven wrote:

Hi Testsuo,

On Mon, Aug 30, 2021 at 4:27 AM Tetsuo Handa
[off-list ref] wrote:

quoted

On 2021/08/30 9:24, Randy Dunlap wrote:

quoted

Note that yres_virtual is set to 0x10000000. Is there no practical limit
(hence limit check) that can be used here?

Also, in vga16fb_check_var(), beginning at line 404:

  404        if (yres > vyres)
  405            vyres = yres;
  406        if (vxres * vyres > maxmem) {
  407            vyres = maxmem / vxres;
  408            if (vyres < yres)
  409                return -ENOMEM;
  410        }

At line 406, the product of vxres * vyres overflows 32 bits (is 0 in this
case/example), so any protection from this block is lost.

OK. Then, we can check overflow like below.

diff --git a/drivers/video/fbdev/vga16fb.c b/drivers/video/fbdev/vga16fb.c
index e2757ff1c23d..e483a3f5fd47 100644
--- a/drivers/video/fbdev/vga16fb.c
+++ b/drivers/video/fbdev/vga16fb.c

@@ -403,7 +403,7 @@ static int vga16fb_check_var(struct fb_var_screeninfo *var,

        if (yres > vyres)
                vyres = yres;
-       if (vxres * vyres > maxmem) {
+       if ((u64) vxres * vyres > (u64) maxmem) {

Mindlessly changing the sizes is not the solution.
Please use e.g. the array_size() helper from <linux/overflow.h>
instead.

On a 64bit system the array_size() macro is going to do the exact same
casts?  But I do think this code would be easier to understand if the
integer overflow check were pull out separately and done first:

	if (array_size(vxres, vyres) >= UINT_MAX)
		return -EINVAL;

	if (vxres * vyres > maxmem) {
		...

The UINT_MAX is because vxres and vyres are u32.

This would maybe be the first time anyone ever did an integer overflow
check like this in the kernel.  It's a new idiom.

regards,
dan carpenter