[PATCH 0/3] gitk: memory consumption improvements

List, Paul,

after searching for a while on why Gitk sometimes consumes exorbitant amounts of memory I found a pair of minor issues and also a big one: the text widget comes with an unlimited undo manager, which is turned on be default. Considering that each line is inserted seperately, this piles up a huuuge undo stack ... for a read-only text widget. Simply turning off this undo manager saves about 95% of memory when viewing large commits (with tens of thousands of diff lines).

3 patches are about to follow:

 - turn off the undo manager,

 - forget already closed file descriptors and

 - forget the 'commitinfo' array on a reload to enforce reloading it.

I hope this finds you appreciation.


Markus

-- 
- - - - - - - - - - - - - - - - - - -
Dipl. Ing. (FH) Markus Hitter
http://www.jump-ing.de/

From e965e1deb9747bbc2b40dc2de95afb65aee9f7fd Mon Sep 17 00:00:00 2001
From: Markus Hitter <redacted>
Date: Sun, 6 Nov 2016 20:38:03 +0100
Subject: [PATCH 1/3] gitk: turn off undo manager in the text widget

The diff text widget is read-only, so there's zero point in
building an undo stack. This change reduces memory consumption of
this widget by about 95%.

Memory usage of the whole program for viewing a reference commit
before; 579'692'744 bytes, after: 32'724'446 bytes.

Test procedure:

 - Choose a largish commit and check it out. In this case one with
   90'802 lines, 5'006'902 bytes.

 - Have a Tcl version with memory debugging enabled. This is,
   build one with --enable-symbols=mem passed to configure.

 - Instrument Gitk to regularly show a memory dump. E.g. by adding
   these code lines at the very bottom:

     proc memDump {} {
         catch {
             set output [memory info]
             puts $output
         }

         after 3000 memDump
     }

     memDump

 - Start Gitk, it'll load this largish commit into the diff text
   field automatically (because it's the current commit).

 - Wait until memory consumption levels out and note the numbers.

Note that the numbers reported by [memory info] are much smaller
than the ones reported in 'top' (1.75 GB vs. 105 MB in this case),
likely due to all the instrumentation coming with the debug
version of Tcl.

Signed-off-by: Markus Hitter <redacted>
---
 gitk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gitk b/gitk
index 805a1c7..8654e29 100755
--- a/gitk
+++ b/gitk

@@ -2403,7 +2403,7 @@ proc makewindow {} {
 
     set ctext .bleft.bottom.ctext
     text $ctext -background $bgcolor -foreground $fgcolor \
-	-state disabled -font textfont \
+	-state disabled -undo 0 -font textfont \
 	-yscrollcommand scrolltext -wrap none \
 	-xscrollcommand ".bleft.bottom.sbhorizontal set"
     if {$have_tk85} {

-- 
2.9.3

From 0a463fcd977dc9558835c373e24a095e35ca3c82 Mon Sep 17 00:00:00 2001
From: Markus Hitter <redacted>
Date: Mon, 7 Nov 2016 16:01:17 +0100
Subject: [PATCH 2/3] gitk: remove closed file descriptors from $blobdifffd

One shouldn't have descriptors of already closed files around.

The first idea to deal with this (previously) ever growing array
was to remove it entirely, but it's needed to detect start of a
new diff with ths old diff not yet done. This happens when a user
clicks on the same commit in the commit list repeatedly without
delay.

Signed-off-by: Markus Hitter <redacted>
---
 gitk | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/gitk b/gitk
index 8654e29..518a4ce 100755
--- a/gitk
+++ b/gitk

@@ -8069,7 +8069,11 @@ proc getblobdiffline {bdf ids} {
     $ctext conf -state normal
     while {[incr nr] <= 1000 && [gets $bdf line] >= 0} {
 	if {$ids != $diffids || $bdf != $blobdifffd($ids)} {
+	    # Older diff read. Abort it.
 	    catch {close $bdf}
+	    if {$ids != $diffids} {
+		array unset blobdifffd $ids
+	    }
 	    return 0
 	}
 	parseblobdiffline $ids $line

@@ -8078,6 +8082,7 @@ proc getblobdiffline {bdf ids} {
     blobdiffmaybeseehere [eof $bdf]
     if {[eof $bdf]} {
 	catch {close $bdf}
+	array unset blobdifffd $ids
 	return 0
     }
     return [expr {$nr >= 1000? 2: 1}]

-- 
2.9.3

From 8359452f426c68cc02250f25f20eaaacd2ddd001 Mon Sep 17 00:00:00 2001
From: Markus Hitter <redacted>
Date: Mon, 7 Nov 2016 19:02:51 +0100
Subject: [PATCH 3/3] gitk: clear array 'commitinfo' on reload

After a reload we might have an entirely different set of commits,
so keeping all of them leaks memory. Remove them all because
re-creating them is not more expensive than testing wether they're
still valid. Lazy (re-)creation is already well established, so
a missing entry can't cause harm.

Signed-off-by: Markus Hitter <redacted>
---
 gitk | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/gitk b/gitk
index 518a4ce..aef6db6 100755
--- a/gitk
+++ b/gitk

@@ -588,7 +588,7 @@ proc updatecommits {} {
 proc reloadcommits {} {
     global curview viewcomplete selectedline currentid thickerline
     global showneartags treediffs commitinterest cached_commitrow
-    global targetid
+    global targetid commitinfo
 
     set selid {}
     if {$selectedline ne {}} {

@@ -609,6 +609,7 @@ proc reloadcommits {} {
 	getallcommits
     }
     clear_display
+    unset -nocomplain commitinfo
     unset -nocomplain commitinterest
     unset -nocomplain cached_commitrow
     unset -nocomplain targetid

-- 
2.9.3

On Mon, Nov 7, 2016 at 10:57 AM, Markus Hitter [off-list ref] wrote:

From e965e1deb9747bbc2b40dc2de95afb65aee9f7fd Mon Sep 17 00:00:00 2001
From: Markus Hitter <redacted>
Date: Sun, 6 Nov 2016 20:38:03 +0100
Subject: [PATCH 1/3] gitk: turn off undo manager in the text widget

The diff text widget is read-only, so there's zero point in
building an undo stack. This change reduces memory consumption of
this widget by about 95%.

Memory usage of the whole program for viewing a reference commit
before; 579'692'744 bytes, after: 32'724'446 bytes.

Wow. Nice find!

Test procedure:

 - Choose a largish commit and check it out. In this case one with
   90'802 lines, 5'006'902 bytes.

 - Have a Tcl version with memory debugging enabled. This is,
   build one with --enable-symbols=mem passed to configure.

 - Instrument Gitk to regularly show a memory dump. E.g. by adding
   these code lines at the very bottom:

     proc memDump {} {
         catch {
             set output [memory info]
             puts $output
         }

         after 3000 memDump
     }

     memDump

 - Start Gitk, it'll load this largish commit into the diff text
   field automatically (because it's the current commit).

 - Wait until memory consumption levels out and note the numbers.

Note that the numbers reported by [memory info] are much smaller
than the ones reported in 'top' (1.75 GB vs. 105 MB in this case),
likely due to all the instrumentation coming with the debug
version of Tcl.

Still, this is definitely the lions share of the memory issue.
Additionally, this fix seems much better overall and does not harm any
other aspects of gitk, because we only read the widget so there is as
you mentioned, zero reason to build an undo stack.

Thanks for taking the extra time to find a proper solution to this! I
think it makes perfect sense.

Nice that such a simple change results in a huge gain. I think this
makes perfect sense.

Regards,
Jake

On Mon, Nov 07, 2016 at 07:54:28PM +0100, Markus Hitter wrote:

List, Paul,

after searching for a while on why Gitk sometimes consumes exorbitant amounts of memory I found a pair of minor issues and also a big one: the text widget comes with an unlimited undo manager, which is turned on be default. Considering that each line is inserted seperately, this piles up a huuuge undo stack ... for a read-only text widget. Simply turning off this undo manager saves about 95% of memory when viewing large commits (with tens of thousands of diff lines).

3 patches are about to follow:

 - turn off the undo manager,

 - forget already closed file descriptors and

 - forget the 'commitinfo' array on a reload to enforce reloading it.

I hope this finds you appreciation.

Thanks for the good work in tracking this down and making the patches.
I have applied the series.  Apologies for slow response (life has been
extremely busy for me this year).

Paul.