On Wed, 27 Sep 2023, Bagas Sanjaya wrote:

On Tue, Sep 26, 2023 at 12:47:51PM -0400, Yaroslav Halchenko wrote:

quoted

Dear Git Gurus,

quoted

In DataLad (https://datalad.org) we are doing lots of automated cloning,
fetching etc as part of our CI etc jobs.  Once in a while git operations
fail [see e.g. 1], and beg us to retry but we need to know when to
do so, and not do it upon every failed git invocation since some
failures could be legit (repository is gone). While looking how others
solve it we found
https://stackoverflow.com/questions/35014012/git-retry-if-http-request-failed
which pointed to tools like git-retry and later part of
https://chromium.googlesource.com/infra/infra/+/HEAD/go/src/infra/tools/git/retry_regexp.go
which serve as a collection of regexes to be on lookout for to retry.

quoted

Would that be the "best" strategy currently?  

Looking at the actual git_retry.py script [1], it really just wraps
actual Git commands. IMO, git-retry(1) shell script as you mentioned
only calls the python version, which adds another level of indirection
(why not doing it in pure shell?).

My guess would be that it is just easier to code in Python usually for
such cases with a "registry" of hits etc.  But why not just to strip .py
from python script which has shebang already and not require bash
wrapper at all? ;)

AFAIK, to solve the retrying problem, we need to have a way to tell
transport backend (curl/ssh) to resume transfer from the faulty point.

some times it seems not even getting connected (https) entirely and that
(?) leading to error in the caller above, e.g. from
https://github.com/datalad/datalad/issues/7485#issuecomment-1735619755

error: Failed to connect to datasets-tests.datalad.org port 443 after 8291 ms: Couldn't connect to server (curl_result = 28, http_code = 0, sha1 = 3980af8de56946a10ff5c48879e5d6025965d936)\nerror: Unable to find 3980af8de56946a10ff5c48879e5d6025965d936 under ...

quoted

As regex matching might eventually break whenever `git` changes
anything in the output messages, I wondered if there could be a more
robust internal implementation in git itself?  Similarly git-annex has
annex.retry config setting which sets the count of retries for
"retriable" operations.  

Do you use porcelain interfaces instead of plumbing ones?

I would say -- a "mix". 

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
WWW:   http://www.linkedin.com/in/yarik