On 2021/08/30 9:24, Randy Dunlap wrote:

Note that yres_virtual is set to 0x10000000. Is there no practical limit
(hence limit check) that can be used here?

Also, in vga16fb_check_var(), beginning at line 404:

  404        if (yres > vyres)
  405            vyres = yres;
  406        if (vxres * vyres > maxmem) {
  407            vyres = maxmem / vxres;
  408            if (vyres < yres)
  409                return -ENOMEM;
  410        }

At line 406, the product of vxres * vyres overflows 32 bits (is 0 in this
case/example), so any protection from this block is lost.

OK. Then, we can check overflow like below.

diff --git a/drivers/video/fbdev/vga16fb.c b/drivers/video/fbdev/vga16fb.c
index e2757ff1c23d..e483a3f5fd47 100644
--- a/drivers/video/fbdev/vga16fb.c
+++ b/drivers/video/fbdev/vga16fb.c

@@ -403,7 +403,7 @@ static int vga16fb_check_var(struct fb_var_screeninfo *var,
 
 	if (yres > vyres)
 		vyres = yres;
-	if (vxres * vyres > maxmem) {
+	if ((u64) vxres * vyres > (u64) maxmem) {
 		vyres = maxmem / vxres;
 		if (vyres < yres)
 			return -ENOMEM;

But I think we can check overflow in the common code like below. (Both patch fixed the oops.)

diff --git a/drivers/video/fbdev/core/fbmem.c b/drivers/video/fbdev/core/fbmem.c
index 1c855145711b..8899679bbc46 100644
--- a/drivers/video/fbdev/core/fbmem.c
+++ b/drivers/video/fbdev/core/fbmem.c

@@ -1008,6 +1008,11 @@ fb_set_var(struct fb_info *info, struct fb_var_screeninfo *var)
 	if (var->xres < 8 || var->yres < 8)
 		return -EINVAL;
 
+	/* Don't allow u32 * u32 to overflow. */
+	if ((u64) var->xres * var->yres > (u64) UINT_MAX ||
+	    (u64) var->xres_virtual * var->yres_virtual > (u64) UINT_MAX)
+		return -EINVAL;
+
 	ret = info->fbops->fb_check_var(var, info);
 
 	if (ret)

But even if yres_virtual (aka vyres) is "only" 0x01000000, so no
multiplication overflow occurs, the resulting value of vyres "seems"
to still be too large and can cause an error [I'm not sure about this
last part -- I need to use a new gcc so that KASAN will work.]