[PATCH] git-http-fetch: Allow caching of retrieved objects by proxy servers

By default the curl library adds "Pragma: no-cache" header to all
requests, which disables caching by proxy servers.  However, most
files in a GIT repository are immutable, and caching them is safe and
could be useful.

This patch removes the "Pragma: no-cache" header from requests for all
files except the pack list (objects/info/packs) and references
(refs/*), which are really mutable and should not be cached.

Signed-off-by: Sergey Vlasov <redacted>


---

 http-fetch.c |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)

fd3f39120636d1ef8834845aa80475c1664a3a3e

diff --git a/http-fetch.c b/http-fetch.c
--- a/http-fetch.c
+++ b/http-fetch.c

@@ -14,6 +14,7 @@
 #endif
 
 static CURL *curl;
+static struct curl_slist *no_pragma_header;
 
 static char *base;
 

@@ -102,6 +103,7 @@ static int fetch_index(unsigned char *sh
 	curl_easy_setopt(curl, CURLOPT_FILE, indexfile);
 	curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite);
 	curl_easy_setopt(curl, CURLOPT_URL, url);
+	curl_easy_setopt(curl, CURLOPT_HTTPHEADER, no_pragma_header);
 	
 	if (curl_easy_perform(curl)) {
 		fclose(indexfile);

@@ -152,6 +154,7 @@ static int fetch_indices(void)
 	curl_easy_setopt(curl, CURLOPT_FILE, &buffer);
 	curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);
 	curl_easy_setopt(curl, CURLOPT_URL, url);
+	curl_easy_setopt(curl, CURLOPT_HTTPHEADER, NULL);
 	
 	if (curl_easy_perform(curl)) {
 		return error("Unable to get pack index %s", url);

@@ -215,6 +218,7 @@ static int fetch_pack(unsigned char *sha
 	curl_easy_setopt(curl, CURLOPT_FILE, packfile);
 	curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite);
 	curl_easy_setopt(curl, CURLOPT_URL, url);
+	curl_easy_setopt(curl, CURLOPT_HTTPHEADER, no_pragma_header);
 	
 	if (curl_easy_perform(curl)) {
 		fclose(packfile);

@@ -255,6 +259,7 @@ int fetch(unsigned char *sha1)
 	curl_easy_setopt(curl, CURLOPT_FAILONERROR, 1);
 	curl_easy_setopt(curl, CURLOPT_FILE, NULL);
 	curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);
+	curl_easy_setopt(curl, CURLOPT_HTTPHEADER, no_pragma_header);
 
 	url = xmalloc(strlen(base) + 50);
 	strcpy(url, base);

@@ -303,6 +308,7 @@ int fetch_ref(char *ref, unsigned char *
         
         curl_easy_setopt(curl, CURLOPT_FILE, &buffer);
         curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);
+	curl_easy_setopt(curl, CURLOPT_HTTPHEADER, NULL);
 
         url = xmalloc(strlen(base) + 6 + strlen(ref));
         strcpy(url, base);

@@ -354,6 +360,7 @@ int main(int argc, char **argv)
 	curl_global_init(CURL_GLOBAL_ALL);
 
 	curl = curl_easy_init();
+	no_pragma_header = curl_slist_append(no_pragma_header, "Pragma:");
 
 	curl_ssl_verify = getenv("GIT_SSL_NO_VERIFY") ? 0 : 1;
 	curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, curl_ssl_verify);

@@ -366,6 +373,7 @@ int main(int argc, char **argv)
 	if (pull(commit_id))
 		return 1;
 
+	curl_slist_free_all(no_pragma_header);
 	curl_global_cleanup();
 	return 0;
 }

Sergey Vlasov wrote:

By default the curl library adds "Pragma: no-cache" header to all
requests, which disables caching by proxy servers.  However, most
files in a GIT repository are immutable, and caching them is safe and
could be useful.

Is caching really safe? Because of compression one git object can have 
many file representations.

Regards
	Stephan

On Wed, 14 Sep 2005, sf wrote:

Sergey Vlasov wrote:

quoted

By default the curl library adds "Pragma: no-cache" header to all
requests, which disables caching by proxy servers.  However, most
files in a GIT repository are immutable, and caching them is safe and
could be useful.

Is caching really safe? Because of compression one git object can have many 
file representations.

only if you use different compression algorithums.

remember that git objects are identified by their sha1, if the sha1 is 
what you want (and the file matches the sha1 after you decompress it) then 
it really doesn't matter what it's on-disk representation is.

David Lang

-- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
  -- C.A.R. Hoare

David Lang wrote:

On Wed, 14 Sep 2005, sf wrote:

quoted

Sergey Vlasov wrote:

quoted

By default the curl library adds "Pragma: no-cache" header to all
requests, which disables caching by proxy servers.  However, most
files in a GIT repository are immutable, and caching them is safe and
could be useful.

Is caching really safe? Because of compression one git object can have 
many file representations.

only if you use different compression algorithums.

Even different implementations and different compression levels can lead 
to different file representations.

remember that git objects are identified by their sha1, if the sha1 is 
what you want (and the file matches the sha1 after you decompress it) 
then it really doesn't matter what it's on-disk representation is.

You are arguing on the git tool level but we are talking about HTTP 
which knows nothing about the uncompressed sha1.

The OP assumed that "files in a GIT repository are immutable" which is 
not true. If you consider the sequence

pack -> prune -> update zlib or git -> unpack

you can end up with different files if the new zlib implementation 
changes imcompatibly (with respect to byte-by-byte compression results) 
or if git suddenly does not use the default compression level any more.

And surely in the future there will be other git implementations than 
this one which may not even use zlib.

I do not say that caching is not possible at all but HTTP caching has 
its pitfalls so just be careful.

Regards

	Stephan

Dear diary, on Mon, Sep 19, 2005 at 12:24:45PM CEST, I got a letter
where sf [off-list ref] told me that...

quoted

remember that git objects are identified by their sha1, if the sha1 is 
what you want (and the file matches the sha1 after you decompress it) 
then it really doesn't matter what it's on-disk representation is.

You are arguing on the git tool level but we are talking about HTTP 
which knows nothing about the uncompressed sha1.

The OP assumed that "files in a GIT repository are immutable" which is 
not true. If you consider the sequence

pack -> prune -> update zlib or git -> unpack

you can end up with different files if the new zlib implementation 
changes imcompatibly (with respect to byte-by-byte compression results) 
or if git suddenly does not use the default compression level any more.

Yes, but why should this matter? It shouldn't matter if you get the old
"version" or the new version of the file over HTTP, the actual object's
contents is still the same, and GIT shouldn't care.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
If you want the holes in your knowledge showing up try teaching
someone.  -- Alan Cox

Petr Baudis wrote:

Dear diary, on Mon, Sep 19, 2005 at 12:24:45PM CEST, I got a letter
where sf [off-list ref] told me that...

...

quoted

The OP assumed that "files in a GIT repository are immutable" which is 
not true. If you consider the sequence

pack -> prune -> update zlib or git -> unpack

you can end up with different files if the new zlib implementation 
changes imcompatibly (with respect to byte-by-byte compression results) 
or if git suddenly does not use the default compression level any more.

Yes, but why should this matter? It shouldn't matter if you get the old
"version" or the new version of the file over HTTP, the actual object's
contents is still the same, and GIT shouldn't care.

This is correct as long as you take care to always get each file in one go.

Recently there was talk about how git handles objects larger than 4GB. 
But you do not have to go this far. Think about fetching 1MB (or 10MB or 
100MB) compressed objects over a slow link. If the transfer gets 
interrupted some people or some clever piece of software - perhaps even 
in git-core - might try to continue the interrupted download. Now if the 
file representation has changed in the meantime the downloaded file is 
going to be corrupt.

The git tools will of course take note of the corruption but then the 
head scratching begins: "What went wrong?"

The more I think about this I realize that my worries have nothing to do 
with caching but with HTTP fetching in general.

Regards

	Stephan