Re: [PATCH v2] convert: add "status=delayed" to filter process protocol

On Mon, Feb 27, 2017 at 11:32:47AM +0100, Lars Schneider wrote:

I completely agree - I need to change that. However, the goal of the v2
iteration was to get the "convert" interface in an acceptable state.
That's what I intended to say in the patch comment section:

    "Please ignore all changes behind async_convert_to_working_tree() and 
     async_filter_finish() for now as I plan to change the implementation 
     as soon as the interface is in an acceptable state."

Ah, sorry, I missed that. I would think the underlying approach would
influence the interface to some degree. But as long as the interface
is sufficiently abstract, I think it gives you enough flexibility.

quoted

From Git's side, the loop is something like:

 while (delayed_items > 0) {
	/* issue a wait, and get back the status/index pair */
	status = send_wait(&index);
	delayed_items--;

	/*
	 * use "index" to find the right item in our list of files;
	 * the format can be opaque to the filter, so we could index
	 * it however we like. But probably numeric indices in an array
	 * are the simplest.
	 */
	assert(index > 0 && index < nr_items);
	item[index].status = status;
	if (status == SUCCESS)
		read_content(&item[index]);
 }

and the filter side just attaches the "index" string to whatever its
internal queue structure is, and feeds it back verbatim when processing
that item finishes.

That could work! I had something like that in mind:

I teach Git a new command "list_completed" or similar. The filter
blocks this call until at least one item is ready for Git. 
Then the filter responds with a list of paths that identify the
"ready items". Then Git asks for these ready items just with the
path and not with any content. Could that work? Wouldn't the path
be "unique" to identify a blob per filter run?

I think that could work, though I think there are few minor downsides
compared to what I wrote above:

  - if you respond with "these items are ready", and then make Git ask
    for each again, it's an extra round-trip for each set of ready
    items. You could just say "an item is ready; here it is" in a single
    response. For a local pipe the latency is probably negligible,
    though.

  - using paths as the index would probably work, but it means Git has
    to use the path to find the "struct checkout_entry" again. Which
    might mean a hashmap (though if you have them all in a sorted list,
    I guess you could also do a binary search).

  - Using an explicit index communicates to the filter not only what the
    index is, but also that Git is prepared to accept a delayed response
    for the item. For backwards compatibility, the filter would probably
    advertise "I have the 'delayed' capability", and then Git could
    choose to use it or not on a per-item basis. Realistically it would
    not change from item to item, but rather operation to operation. So
    that means we can easily convert the call-sites in Git to the async
    approach incrementally. As each one is converted, it turns on the
    flag that causes the filter code to send the "index" tag.

-Peff