Re: [PATCH v1] convert: add "status=delayed" to filter process protocol

On 10 Jan 2017, at 23:11, Jakub Narębski [off-list ref] wrote:

W dniu 09.01.2017 o 00:42, Junio C Hamano pisze:

quoted

larsxschneider@gmail.com writes:

quoted

From: Lars Schneider <redacted>

Some `clean` / `smudge` filters might require a significant amount of
time to process a single blob. During this process the Git checkout
operation is blocked and Git needs to wait until the filter is done to
continue with the checkout.

Lars, what is expected use case for this feature; that is when do you
think this problem may happen?  Is it something that happened IRL?

Yes, this problem happens every day with filters that perform network
requests (e.g. GitLFS). In GitLFS we even implemented Git wrapper
commands to address the problem: https://github.com/git-lfs/git-lfs/pull/988
The ultimate goal of this patch is to be able to get rid of the wrapper 
commands.

quoted

quoted

Teach the filter process protocol (introduced in edcc858) to accept the
status "delayed" as response to a filter request. Upon this response Git
continues with the checkout operation and asks the filter to process the
blob again after all other blobs have been processed.

Hmm, I would have expected that the basic flow would become

	for each paths to be processed:
		convert-to-worktree to buf
		if not delayed:
			do the caller's thing to use buf
		else:
			remember path

	for each delayed paths:
		ensure filter process finished processing for path
		fetch the thing to buf from the process
		do the caller's thing to use buf

I would expect here to have a kind of event loop, namely

       while there are delayed paths:
               get path that is ready from filter
               fetch the thing to buf (supporting "delayed")
               if path done
                       do the caller's thing to use buf 
                       (e.g. finish checkout path, eof convert, etc.)

We can either trust filter process to tell us when it finished sending
delayed paths, or keep list of paths that are being delayed in Git.

I could implement "get path that is ready from filter" but wouldn't
that complicate the filter protocol? I think we can use the protocol pretty 
much as if with the strategy outlined here:
http://public-inbox.org/git/F533857D-9B51-44C1-8889-AA0542AD8250@gmail.com/


Thanks,
Lars