RE: High locking contention during repack?

(Apologize for top-posting but making Outlook quote properly is a bear)

Down to 1+4 threads:

--total-cpu-usage-- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai stl| read  writ| recv  send|  in   out | int   csw 
  0  13  87   0   0|   0     0 |2059B   15k|   0     0 |2200  1730 
  0  13  87   0   0|   0     0 | 585B 2754B|   0     0 |1792  1269 
  0  13  87   0   0|   0     0 | 178B  734B|   0     0 |1691  1157 
  0  13  87   0   0|   0  6554B| 225B  734B|   0     0 |1885  1238 
  0  13  87   0   0|   0     0 | 159B  736B|   0     0 |1958  1262 
  0  13  87   0   0|   0     0 | 310B  760B|   0     0 |1813  1175

(gdb) info threads
  Id   Target Id         Frame 
* 1    Thread 0x7f7b837dcb80 (LWP 11918) "git" 0x000055c7dded56eb in ll_find_deltas (processed=0x7ffce7da82a8, depth=<optimized out>, window=<optimized out>, 
    list_size=0, list=<optimized out>) at builtin/pack-objects.c:2516
  2    Thread 0x7f782e16a700 (LWP 11929) "git" create_delta (index=0x55c807d2ff20, trg_buf=<optimized out>, trg_size=trg_size@entry=9677636, 
    delta_size=delta_size@entry=0x7f782e169e78, max_size=3585674) at diff-delta.c:381
  3    Thread 0x7f6fe77fe700 (LWP 11948) "git" create_delta (index=0x7f6e35171010, trg_buf=<optimized out>, trg_size=trg_size@entry=163385853, 
    delta_size=delta_size@entry=0x7f6fe77fde78, max_size=2328926) at diff-delta.c:381
  4    Thread 0x7f6fe67fc700 (LWP 11950) "git" create_delta (index=0x7f6e3f7f3010, trg_buf=<optimized out>, trg_size=trg_size@entry=74543780, 
    delta_size=delta_size@entry=0x7f6fe67fbe78, max_size=2582505) at diff-delta.c:381
  5    Thread 0x7f6fe5ffb700 (LWP 11951) "git" create_delta (index=0x7f6f01d06010, trg_buf=<optimized out>, trg_size=trg_size@entry=28510185, 
    delta_size=delta_size@entry=0x7f6fe5ffae78, max_size=9146484) at diff-delta.c:381
(gdb) bt
#0  0x00007f7b82f909f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55c7de2ff348 <progress_cond+40>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55c7de2ff360 <progress_mutex>, cond=0x55c7de2ff320 <progress_cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=cond@entry=0x55c7de2ff320 <progress_cond>, mutex=mutex@entry=0x55c7de2ff360 <progress_mutex>) at pthread_cond_wait.c:655
#3  0x000055c7dded56eb in ll_find_deltas (processed=0x7ffce7da82a8, depth=<optimized out>, window=<optimized out>, list_size=0, list=<optimized out>)
    at builtin/pack-objects.c:2516
#4  prepare_pack (depth=<optimized out>, window=<optimized out>) at builtin/pack-objects.c:2673
#5  cmd_pack_objects (argc=<optimized out>, argv=<optimized out>, prefix=<optimized out>) at builtin/pack-objects.c:3491
#6  0x000055c7dde74711 in run_builtin (argv=<optimized out>, argc=<optimized out>, p=<optimized out>) at git.c:421
#7  handle_builtin (argc=<optimized out>, argv=<optimized out>) at git.c:647
#8  0x000055c7dde75725 in run_argv (argv=0x7ffce7daa7e0, argcp=0x7ffce7daa7ec) at git.c:701
#9  cmd_main (argc=<optimized out>, argv=<optimized out>) at git.c:798
#10 0x000055c7dde7437f in main (argc=6, argv=0x7ffce7daaa58) at common-main.c:45
(gdb) thread 2
[Switching to thread 2 (Thread 0x7f782e16a700 (LWP 11929))]
#0  create_delta (index=0x55c807d2ff20, trg_buf=<optimized out>, trg_size=trg_size@entry=9677636, delta_size=delta_size@entry=0x7f782e169e78, max_size=3585674)
    at diff-delta.c:381
381                                     if (entry->val != val)
(gdb) bt
#0  create_delta (index=0x55c807d2ff20, trg_buf=<optimized out>, trg_size=trg_size@entry=9677636, delta_size=delta_size@entry=0x7f782e169e78, max_size=3585674)
    at diff-delta.c:381
#1  0x000055c7dded0606 in try_delta (mem_usage=<synthetic pointer>, max_depth=50, src=0x7f702c256bc0, trg=0x7f702c256ca0) at builtin/pack-objects.c:2129
#2  find_deltas (list=<optimized out>, list_size=list_size@entry=0x55c7e467daf4, window=<optimized out>, depth=<optimized out>, processed=<optimized out>)
    at builtin/pack-objects.c:2262
#3  0x000055c7dded0e29 in threaded_find_deltas (arg=0x55c7e467dae0) at builtin/pack-objects.c:2408
#4  0x00007f7b82f8a6db in start_thread (arg=0x7f782e16a700) at pthread_create.c:463
#5  0x00007f7b82cb388f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) thread 2
[Switching to thread 2 (Thread 0x7f782e16a700 (LWP 11929))]
#0  create_delta (index=0x55c807d2ff20, trg_buf=<optimized out>, trg_size=trg_size@entry=9677636, delta_size=delta_size@entry=0x7f782e169e78, max_size=3585674)
    at diff-delta.c:381
381                                     if (entry->val != val)
(gdb) bt
#0  create_delta (index=0x55c807d2ff20, trg_buf=<optimized out>, trg_size=trg_size@entry=9677636, delta_size=delta_size@entry=0x7f782e169e78, max_size=3585674)
    at diff-delta.c:381
#1  0x000055c7dded0606 in try_delta (mem_usage=<synthetic pointer>, max_depth=50, src=0x7f702c256bc0, trg=0x7f702c256ca0) at builtin/pack-objects.c:2129
#2  find_deltas (list=<optimized out>, list_size=list_size@entry=0x55c7e467daf4, window=<optimized out>, depth=<optimized out>, processed=<optimized out>)
    at builtin/pack-objects.c:2262
#3  0x000055c7dded0e29 in threaded_find_deltas (arg=0x55c7e467dae0) at builtin/pack-objects.c:2408
#4  0x00007f7b82f8a6db in start_thread (arg=0x7f782e16a700) at pthread_create.c:463
#5  0x00007f7b82cb388f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

florin

-----Original Message-----
From: Iucha, Florin 
Sent: Wednesday, December 12, 2018 11:50 AM
To: 'Jeff King' <redacted>
Cc: git@vger.kernel.org
Subject: RE: High locking contention during repack?

Jeff,

Thank you for the advice, I will reduce the depth.

Running "git pack-objects --all --no-reuse-delta --delta-base-offset --stdout </dev/null >/dev/null", it got to 99% fairly quickly, now it has 5 threads running (using 99.9% CPU), but the "dstat 5" still shows lots of nothing, with some heavy system activity:

--total-cpu-usage-- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai stl| read  writ| recv  send|  in   out | int   csw 
  1   8  91   0   0| 166k  224k|   0     0 |   0     0 |6892   970 
  0  13  87   0   0|   0     0 | 333B 1758B|   0     0 |2285  1374 
  0  13  87   0   0|   0     0 | 309B 1190B|   0     0 |2233  1265 
  0  13  87   0   0|   0     0 | 282B 1053B|   0     0 |2174  1239 
  0  13  87   0   0|   0     0 | 278B 1251B|   0     0 |1930  1173 
  0  13  87   0   0|   0     0 | 274B  942B|   0     0 |1987  1139 
  0  13  87   0   0|   0     0 | 264B 1168B|   0     0 |1928  1205 
  0  13  87   0   0|   0     0 | 388B 2435B|   0     0 |2245  1280 
  0  13  87   0   0|   0     0 | 268B 1145B|   0     0 |2225  1221 
  0  13  87   0   0|   0     0 | 164B  732B|   0     0 |2607  1333 
  0  13  87   0   0|   0     0 | 156B  996B|   0     0 |2100  1270 
  0  13  87   0   0|   0     0 | 206B 1019B|   0     0 |2192  1296 
  0  13  87   0   0|   0     0 | 198B  824B|   0     0 |2019  1236 
  0  13  87   0   0|   0     0 | 245B  435B|   0     0 |1974  1195 
  0  13  87   0   0|   0     0 | 252B  855B|   0     0 |1852  1166 
  0  13  87   0   0|   0     0 | 230B  758B|   0     0 |2066  1299 
  0  13  87   0   0|   0     0 | 284B  925B|   0     0 |1860  1225 
  0  13  87   0   0|   0     0 | 289B 2682B|   0     0 |1796  1197 
  0  13  87   0   0|   0     0 | 939B 1263B|   0     0 |1913  1304 
  0  13  87   0   0|   0     0 |1212B 1366B|   0     0 |1915  1343

I will try running the command under the debugger and stop it when it gets to this point and poke around.

florin