[PATCH] git-daemon server

DORMANTno replies

18 messages, 5 authors, 2016-06-15 · open the first message on its own page

[PATCH] git-daemon server

From: Jason McMullan <hidden>
Date: 2016-06-15 22:41:59

git-daemon - Client/server for exchanging GIT objects

Requires: [PATCH] Expose more sha1_file.c interfaces
Requires: [PATCH] Anal retentive 'const unsigned char *sha1'
Signed-off-by: Jason McMullan <redacted>

GIT daemon is a symmetric protocol for exchanging objects
between repositories. It has the following features:

	* Can use TCP, Unix domain sockets, or stdin/stdout
	* Verifies received SHA1 objects are valid and fully
	  connected before they are stored in the object database,
	  including delta objects.
	* Fully symmetric protocol
	* Only pulls the dependencies of the requested object -
	  allows you to place *multiple* projects in one GIT database.
	* Proper head locking, allows either lazy or cmp-xchg updates
	  (lazy is just 'head head-id old-sha1 new-sha1', cmp-xchg is
           'send new-sha1; head head-id old-sha1 new-sha1' )

TODO:
-----

	* Command line 'send' of a SHA1 from the local database
	  (for the cmd-xchg mechanism. Lazy works out-of-the box)
	* Optional auto-deltification of transmitted objects
	* More documentation
	* Test cases for 'make test'
	* Merge verify-before-update logic into pull.c

GIT Daemon Protocol Commands:
-----------------------------

version::
        Requests the version string, returned in an error message
        Expects: 'error'

head <head-id>::
        Requests SHA1 of the <head-id>
        Expects: 'sha1' or 'error'

head <head-id> <old-sha1> <new-sha1>::
        Requests that <head-id> be updated with <new-sha1>. Use
        '--' for <old-sha1> to create a new head.
        Expects: 'sha1', 'request', or 'error'

exists <sha1>::
        Determine if SHA1 exists in remote
        Expects: 'sha1' or 'error'

request <sha1>::
        Request SHA1 object. 
        Expects: 'send' or 'error'

send <sha1>::
        Send SHA1 object.
        Expects: 'sha1' or 'error'

error <key> <hex-code> <error-message>::
        Returned error code
        Expects: Nothing

sha1 <key> <hex-sha1>::
        Success message for 'head' and 'send' commands.


diff -u b/daemon.c b/daemon.c
--- b/daemon.c
+++ b/daemon.c
@@ -0,0 +1,827 @@
+#include <stdio.h>
+#include <fcntl.h>
+#include <unistd.h>
+#include <ctype.h>
+#include <assert.h>
+#include <signal.h>
+#include <sys/un.h>
+#include <sys/file.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+#include <sys/socket.h>
+#include <netinet/in.h>
+#include <netinet/ip.h>
+#include <arpa/inet.h>
+#include <netdb.h>
+
+#ifndef UNIX_PATH_MAX
+#define UNIX_PATH_MAX sizeof(((struct sockaddr_un *)NULL)->sun_path)
+#endif
+
+#include "cache.h"
+#include "delta.h"
+#include "object.h"
+#include "blob.h"
+#include "tree.h"
+#include "commit.h"
+#include "tag.h"
+
+#define ERROR_VERSION			0,"GIT Daemon protocol version 0.0"
+#define ERROR_INVALID			1,"Invalid command"
+#define ERROR_REQUEST_INVALID		100,"SHA1 does not exist"
+#define ERROR_HEAD_BAD(hash)		200,"Head is not at hash %s",hash
+#define ERROR_HEAD_INVALID		201,"Head does not exist"
+#define ERROR_HEAD_LOCK			202,"Head can't be locked"
+#define ERROR_HEAD_UPDATE		203,"Head can't be updated"
+#define ERROR_HEAD_SHA1_INVALID(sha1)	204,"SHA1 %s does not exist",sha1
+#define ERROR_SEND_INVALID		300,"Invalid data sent on send"
+
+/* Protocol is symmetric, both client and server
+ * use the same commands.
+ *
+ * version\n -> error -- 0 <Version string>\n
+ *
+ * head <head-id>\n -> sha1 <head-id> <sha1>\n
+ *
+ * head <head-id> <old-sha1> <new-sha1>\n -> sha1 <head-id> <new-sha1>\n
+ *
+ * request <sha1>\n -> send <sha1> <hex-bytes>\n<bytes...>
+ *
+ * send <sha1> <hex-bytes>\n<bytes...> -> sha1 -- <sha1>\n
+ *
+ * exists <sha1>\n -> sha1 -- <sha1>\n
+ *
+ * sha1 <any> <sha1>\n -> no-op
+ *
+ * error <key> <hex-code> <error string>\n -> no-op
+ *
+ */
+
+static int fd_in=0, fd_out=1;
+
+static int process_request();
+static int send_request(const unsigned char *sha1);
+static const char *git_dir;
+
+void *map_file(int fd, size_t size)
+{
+	void *map;
+
+	if (fd < 0) {
+		error("cannot map file on fd %d", fd);
+		return NULL;
+	}
+
+	map = mmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0);
+	if (-1 == (int)(long)map)
+		return NULL;
+	return map;
+}
+
+static int sha1_retrieve(const unsigned char *sha1)
+{
+	int err;
+
+       	if (!has_sha1_file(sha1)) {
+		err = send_request(sha1);
+		if (err < 0)
+			return err;
+	}
+
+	return 0;
+}
+
+static int verify_file(int fd, unsigned long mapsize, const unsigned char *sha1, char *type)
+{
+	void *map, *buffer;
+	char hdr[8192];
+	struct object *obj;
+	unsigned long size;
+	z_stream stream;
+	int err;
+
+	map = map_file(fd, mapsize);
+	if (map) {
+		err = unpack_sha1_header(&stream, map, mapsize, hdr, sizeof(hdr));
+		if (err < Z_OK || parse_sha1_header(hdr, type, &size) < 0) {
+			munmap(map, mapsize);
+			return -1;
+		}
+		buffer = unpack_sha1_rest(&stream, hdr, size);
+		munmap(map, size);
+
+		if (buffer == NULL) {
+			return -1;
+		}
+
+		if (buffer && !strcmp(type, "delta")) {
+			void *ref = NULL, *delta = buffer;
+			unsigned long ref_size, delta_size = size;
+			buffer = NULL;
+			if (delta_size <= 20) {
+				free(buffer);
+				return -1;
+			}
+			err = sha1_retrieve(delta);
+			if (err < 0) {
+				free(buffer);
+				return err;
+			}
+			ref = read_sha1_file(delta, type, &ref_size);
+
+			if (ref == NULL) {
+				free(buffer);
+				return err;
+			}
+
+			buffer = patch_delta(ref, ref_size,
+					  delta+20, delta_size-20, 
+					  &size);
+			free(delta);
+			free(ref);
+		}
+
+		if (!strcmp(type, "blob")) {
+			struct blob *blob = lookup_blob(sha1);
+			parse_blob_buffer(blob, buffer, size);
+			obj = &blob->object;
+		} else if (!strcmp(type, "tree")) {
+			struct tree *tree = lookup_tree(sha1);
+			parse_tree_buffer(tree, buffer, size);
+			obj = &tree->object;
+		} else if (!strcmp(type, "commit")) {
+			struct commit *commit = lookup_commit(sha1);
+			parse_commit_buffer(commit, buffer, size);
+			if (!commit->buffer) {
+				commit->buffer = buffer;
+				buffer = NULL;
+			}
+			obj = &commit->object;
+		} else if (!strcmp(type, "tag")) {
+			struct tag *tag = lookup_tag(sha1);
+			parse_tag_buffer(tag, buffer, size);
+			obj = &tag->object;
+		} else {
+			obj = NULL;
+		}
+
+		free(buffer);
+
+		if (obj) {
+			struct object_list *refs;
+
+			for (refs = obj->refs; refs ; refs = refs->next) {
+				err = sha1_retrieve(refs->item->sha1);
+				if (err < 0)
+					return err;
+			}
+		
+			return 0;
+		}
+
+	}
+	return -1;
+}
+
+static int send_command(int argc, char **argv)
+{
+	char buff[1024];
+	int len, size;
+
+	size = snprintf(buff, sizeof(buff), "%s", argv[0]);
+	for ( argc--,argv++; argc > 0 ; argc--, argv++) {
+		len = snprintf(buff + size, sizeof(buff) - size, " %s", argv[0]);
+		size += len;
+	}
+	buff[size++]='\n';
+	return write(fd_out, buff, size);
+}
+
+static int send_error(const char *key,int code,const char *errstr,...)
+{
+	va_list args;
+	static char buff[1024];
+	int len;
+
+	va_start(args, errstr);
+
+	len = snprintf(buff, sizeof(buff), "error %s %x ", key ? key : "--", code);
+	write(fd_out, buff, len);
+	len = vsnprintf(buff, sizeof(buff), errstr, args);
+	write(fd_out, buff, len);
+	write(fd_out, "\n", 1);
+
+	va_end(args);
+
+	return 0;
+}
+
+static int send_sha1(const char *key, const unsigned char *sha1)
+{
+	char buff[2048];
+	int len;
+
+	len = snprintf(buff,sizeof(buff), "sha1 %s %s\n",key ? key : "--", sha1_to_hex(sha1));
+	return write(fd_out, buff, len);
+}
+
+static int send_send(const unsigned char *sha1, int size, void *data)
+{
+	char buff[2048];
+	int len, err;
+
+	len = snprintf(buff,sizeof(buff), "send %s %x\n",sha1_to_hex(sha1),size);
+	err = write(fd_out, buff, len);
+
+	if (err < 0)
+		return err;
+
+	err = write(fd_out, data, size);
+	if (err < 0)
+		return err;
+
+	return process_request();
+}
+
+static int send_request(const unsigned char *sha1)
+{
+	char buff[2048];
+	int len, err;
+
+	len = snprintf(buff,sizeof(buff), "request %s\n",sha1_to_hex(sha1));
+	err = write(fd_out, buff, len);
+	if (err < 0)
+		return err;
+
+	return process_request();
+}
+
+
+static inline int lock_file(const char *path, int is_create)
+{
+	int fd;
+
+	fd = open(path, O_RDONLY);
+	if (fd < 0 && is_create) {
+		fd = open(path, O_RDWR | O_CREAT | O_TRUNC, 0644);
+	} else if (is_create) {
+		close(fd);
+		return -1;
+	}
+
+	if (fd < 0)
+		return -1;
+
+	if (flock(fd, LOCK_EX | LOCK_NB) < 0) {
+		close(fd);
+		if (is_create)
+			unlink(path);
+		return -1;
+	}
+
+	return fd;
+}
+
+static inline void unlock_file(int fd)
+{
+	flock(fd, LOCK_UN);
+	close(fd);
+}
+
+static int cmd_version(int argc, char **argv)
+{
+	return send_error("--",ERROR_VERSION);
+}
+
+/* Return < 0 on I/O error
+ */
+static int cmd_head(int argc, char **argv)
+{
+	int len, err;
+
+	if (argc == 2) {
+		unsigned char sha1[20];
+
+		/* Get SHA1 for head */
+	       	err = get_sha1(argv[1], sha1);
+		if (err < 0)
+			return send_error(argv[1],ERROR_HEAD_INVALID);
+
+		return send_sha1(argv[1],sha1);
+	} else if (argc == 4) {
+		char head_sha1[20], sha1[20], sha1_new[20];
+		int fd, head_fd;
+		static char buff[256];
+		static char hfile[PATH_MAX];
+		static char hfile_new[PATH_MAX];
+		int is_create;
+		const char *head=argv[1], *old_hash=argv[2], *new_hash=argv[3];
+
+		is_create = (strcmp(old_hash,"--")==0);
+
+		/* Is the head name valid? */
+		if (strchr(head,'/') != NULL)
+			return send_error(head,ERROR_HEAD_INVALID);
+
+		/* lock head */
+		len = snprintf(hfile, sizeof(hfile), "%s/refs/heads/%s", git_dir, argv[1]);
+		strcpy(hfile_new, hfile);
+		strcpy(hfile_new + len, ".new");
+		head_fd = lock_file(hfile, is_create);
+		if (head_fd < 0) {
+			return send_error(head,ERROR_HEAD_LOCK);
+		}
+
+		/* Verify that we know what we're doing
+		 * to do a cmp-xchg update
+		 */
+		if (!is_create) {
+	       		err = get_sha1(head, head_sha1);
+			if (err < 0) {
+				if (is_create) unlink(hfile);
+				unlock_file(head_fd);
+				return send_error(head, ERROR_HEAD_INVALID);
+			}
+
+			err = get_sha1(old_hash, sha1);
+			if (err < 0) {
+				if (is_create) unlink(hfile);
+				unlock_file(head_fd);
+				return send_error(head, ERROR_HEAD_SHA1_INVALID(old_hash));
+			}
+
+			if (memcmp(head_sha1, sha1, 20) != 0) {
+				if (is_create) unlink(hfile);
+				unlock_file(head_fd);
+				return send_error(head, ERROR_HEAD_BAD(old_hash));
+			}
+		}
+
+
+		/* Is the new head id even a valid string? */
+		err = get_sha1(new_hash, sha1_new);
+		if (err < 0) {
+			if (is_create) unlink(hfile);
+			unlock_file(head_fd);
+			return send_error(head,ERROR_HEAD_SHA1_INVALID(new_hash));
+		}
+
+		/* If new-sha1 == old-sha1, short circuit to end */
+		if (memcmp(sha1_new, sha1, 20) == 0)
+			goto done;
+
+		/* Verify that new-sha1 is in the database */
+		err = sha1_retrieve(sha1_new);
+		if (err < 0) {
+			if (is_create) unlink(hfile);
+			unlock_file(head_fd);
+			return err;
+		}
+
+		/* We now have the SHA1. Update head */
+		fd = open(hfile_new, O_RDWR | O_CREAT | O_TRUNC, 0644);
+		if (fd < 0) {
+			if (is_create) unlink(hfile);
+			unlock_file(head_fd);
+			return send_error(argv[1],ERROR_HEAD_UPDATE);
+		}
+
+		len = snprintf(buff, sizeof(buff), "%s\n", sha1_to_hex(sha1_new));
+		err = write(fd, buff, len);
+		close(fd);
+		if (err != len) {
+			if (is_create) unlink(hfile);
+			unlink(hfile_new);
+			unlock_file(head_fd);
+			return send_error(argv[1],ERROR_HEAD_UPDATE);
+		}
+
+		err = rename(hfile_new, hfile);
+		if (err < 0) {
+			if (is_create) unlink(hfile);
+			unlock_file(head_fd);
+			return send_error(argv[1],ERROR_HEAD_UPDATE);
+		}
+
+done:
+		/* Unlock head */
+		unlock_file(head_fd);
+		return send_sha1(argv[1], sha1_new);
+		write(fd_out, buff, len);
+	} else {
+		return send_error(argv[1],ERROR_INVALID);
+	}
+
+	return 0;
+}
+
+/* Return < 0 on I/O error
+ */
+static int cmd_request(int argc, char **argv)
+{
+	int err;
+
+	if (argc == 2) {
+		char sha1[40];
+		void *data;
+		unsigned long size;
+
+ 		err = get_sha1(argv[1], sha1);
+
+		/* If sha1 exists, send it. */
+		if (err < 0)
+			return send_error(argv[1],ERROR_REQUEST_INVALID);
+
+		data = map_sha1_file(sha1, &size);
+		if (data == NULL)
+			return send_error(argv[1],ERROR_REQUEST_INVALID);
+
+		err = send_send(sha1, size, data);
+		if (err < 0)
+			return err;
+		err = process_request();
+		munmap(data, size);
+	} else {
+		err = send_error(argv[1],ERROR_INVALID);
+	}
+
+	return err;
+}
+
+static int cmd_send(int argc, char **argv)
+{
+	int err;
+	int tmpfd = -1;
+	char tmpn[]="/tmp/git.daemon.XXXXXX";
+
+	if (argc == 3) {
+		char sha1[20];
+		int size = strtol(argv[2],NULL,16);
+		char type[32];
+		void *buff;
+		const char *fname;
+		int fd;
+		int exists = 0;
+
+ 		err = get_sha1(argv[1], sha1);
+		/* If sha1 exists, just drop all the bytes on the floor */
+		/* Otherwise, create a temp file, and stuff the bytes there */
+
+		if (err == 0) {
+			tmpfd = mkstemp(tmpn);
+			unlink(tmpn);
+		} else {
+			exists = 1;
+			tmpfd = open("/dev/null", O_RDWR);
+		}
+
+		err = write_sha1_from_fd_to_fd(sha1, fd_in, tmpfd);
+		if (err < 0 || size != err) {
+			close(tmpfd);
+			return send_error(argv[1],ERROR_SEND_INVALID);
+		}
+
+		if (exists)
+			goto done;
+
+		/* Verify that the temp file contains a valid blob/tree 
+		 * This step also recurses and gets all the 'good stuff'
+		 */
+		err = verify_file(tmpfd, size, sha1, type);
+		if (err < 0) {
+			close(tmpfd);
+			return send_error(argv[1],ERROR_SEND_INVALID);
+		}	
+
+		/* Write to database if verified */
+		buff = map_file(tmpfd, size);
+		fname = sha1_file_name(sha1);
+		fd = open(fname, O_RDWR | O_CREAT | O_TRUNC, 0644);
+		err = write(fd, buff, size);
+		close(fd);
+		if (err != size) {
+			unlink(fname);
+			close(tmpfd);
+			return send_error(argv[1],ERROR_SEND_INVALID);
+		}
+
+done:
+		close(tmpfd);
+		return send_sha1(NULL, sha1);
+	} else {
+		err = send_error(argv[1],ERROR_INVALID);
+	}
+
+	return err;
+}
+
+int cmd_exists(int argc, char **argv)
+{
+	int err;
+
+	if (argc == 2) {
+		char sha1[20];
+
+ 		err = get_sha1(argv[1], sha1);
+
+		/* If sha1 exists, say so. Otherwise say 'send sha1'*/
+		if (err >= 0)
+			return send_sha1(NULL, sha1);
+
+		return send_error(NULL, ERROR_REQUEST_INVALID);
+	} else {
+		err = send_error(argv[1],ERROR_INVALID);
+	}
+
+	return err;
+}
+
+int cmd_sha1(int argc, char **argv)
+{
+	int i;
+
+	fprintf(stderr, "%s",argv[0]);
+	for ( i=1; i < argc; i++)
+		fprintf(stderr, " %s",argv[i]);
+	fprintf(stderr, "\n");
+	return 0;
+}
+
+static int cmd_error(int argc, char **argv)
+{
+	int i;
+
+	fprintf(stderr, "%s",argv[0]);
+	for ( i=1; i < argc; i++)
+		fprintf(stderr, " %s",argv[i]);
+	fprintf(stderr, "\n");
+	return 0;
+}
+
+static int command(int argc, char **argv)
+{
+	int err;
+#if 0
+	int i;
+
+	fprintf(stderr, "%s",argv[0]);
+	for ( i=1; i < argc; i++)
+		fprintf(stderr, " %s",argv[i]);
+	fprintf(stderr, "\n");
+#endif
+
+	if (argc < 1)
+		goto invalid;
+
+	if (strcmp(argv[0],"version")==0) {
+		err = cmd_version(argc,argv);
+		return err;
+	} else if (strcmp(argv[0],"head")==0) {
+		err = cmd_head(argc,argv);
+		return err;
+	} else if (strcmp(argv[0],"send")==0) {
+		err = cmd_send(argc,argv);
+		return err;
+	} else if (strcmp(argv[0],"request")==0) {
+		err = cmd_request(argc,argv);
+		return err;
+	} else if (strcmp(argv[0],"exists")==0) {
+		err = cmd_exists(argc,argv);
+		return err;
+	} else if (strcmp(argv[0],"sha1")==0) {
+		err = cmd_sha1(argc,argv);
+		return err;
+	} else if (strcmp(argv[0],"error")==0) {
+		err = cmd_error(argc,argv);
+		return err;
+	}
+
+invalid:
+	return send_error("--",ERROR_INVALID);
+}
+
+/* This *intentionally* drops requests on the floor that are
+ * larger than sizeof(buffer)
+ */
+static int process_request()
+{
+	char buffer[1024];
+	int offset;
+	char **argv,*cp;
+	int argc,err;
+
+	offset=0;
+
+	while (read(fd_in, &buffer[offset], 1)==1) {
+		if (buffer[offset++]=='\n')
+			break;
+		if (offset == sizeof(buffer))
+			return -1;
+	}
+	offset--;
+
+	if (offset < 0 || buffer[offset]!='\n')
+		return -1;
+
+	buffer[offset]=0;
+	argc=0;
+	argv=xmalloc(sizeof(char *));
+
+	cp = &buffer[0];
+	while (1) {
+		while (*cp && isspace(*cp)) cp++;
+		if (*cp == 0)
+			break;
+		argv[argc++]=cp;
+		argv=xrealloc(argv, (argc+1)*sizeof(char *));
+		while (*cp && !isspace(*cp)) cp++;
+		if (*cp == 0)
+			break;
+		*(cp++) = 0;
+	};
+
+	argv[argc]=NULL;
+	err = command(argc,argv);
+	free(argv);
+
+	return err;
+}
+
+/* Reap children */
+static void reaper(int sig)
+{
+	int status;
+
+	waitpid(-1, &status, WNOHANG);
+}
+
+static int get_ip_addr(const char *ip, struct in_addr *addr)
+{
+	int err;
+	struct hostent *host;
+
+	err = inet_pton(AF_INET, ip, addr);
+	if (err < 0)
+		return 0;
+
+	host = gethostbyname(ip);
+	if (host == NULL)
+		return -1;
+
+	return inet_pton(AF_INET, host->h_addr, addr);
+}
+
+static int get_port(const char *port)
+{
+	struct servent *serv;
+	int p;
+
+	if (sscanf(port,"%d",&p) == 1)
+		return p;
+
+	serv = getservbyname(port,"tcp");
+	if (serv == NULL)
+		return -1;
+
+	return serv->s_port;
+}
+
+int main(int argc, char **argv)
+{
+	enum {
+		MODE_STDIO,
+		MODE_UNIX,
+		MODE_TCP,
+	} mode = MODE_STDIO;
+	const char *port = "/tmp/.gitd",*host="127.0.0.1";
+	union {
+		struct sockaddr	   generic;
+		struct sockaddr_un un;
+		struct sockaddr_in in;
+	} sock;
+	struct sockaddr_un *s_unix = (void *)&sock.un;
+	struct sockaddr_in *s_inet = (void *)&sock.in;
+	size_t s_size = 0;
+	int max_clients=100, err;
+
+	git_dir = gitenv(GIT_DIR_ENVIRONMENT) ? : DEFAULT_GIT_DIR_ENVIRONMENT;
+
+	argv++;
+	argc--;
+	for (; argc > 0; argv++, argc--) {
+		char *path = argv[0];
+
+		if (path[0] != '-')
+		       break;
+
+		if (!strcmp(path,"--")) {
+			argv++;
+			argc--;
+			break;
+		}
+
+		if (!strcmp(path,"--unix")) {
+			mode = MODE_UNIX;
+			continue;
+		}
+		
+		if (!strcmp(path,"--tcp")) {
+			mode = MODE_TCP;
+			continue;
+		}
+		
+		if (!strcmp(path, "--max-clients")) {
+			if (argc < 2)
+				die("git-daemon: --max-clients <number>");
+			max_clients = strtoul(argv[1],NULL,0);
+			argv++;
+			argc--;
+			continue;
+		}
+
+		if (!strcmp(path, "--host")) {
+			if (argc < 2)
+				die("git-daemon: --host <host>");
+			host = argv[1];
+			argv++;
+			argc--;
+			continue;
+		}
+
+		if (!strcmp(path, "--port")) {
+			if (argc < 2)
+				die("git-daemon: --port <port>");
+			port = argv[1];
+			argv++;
+			argc--;
+			continue;
+		}
+
+		if (!strcmp(path, "--db")) {
+			if (argc < 2)
+				die("git-daemon: --db <path/.git>");
+			git_dir = argv[2];
+			argv++;
+			argc--;
+			continue;
+		}
+	}
+
+	switch (mode) {
+		case MODE_STDIO: fd_in = 0; fd_out = 1; break;
+		case MODE_UNIX:
+			fd_in = fd_out = socket(PF_UNIX, SOCK_STREAM, 0);
+			if (fd_in < 0)
+				die("git-daemon: Can't create local domain socket");
+			strncpy(s_unix->sun_path, port, UNIX_PATH_MAX);
+			s_unix->sun_path[UNIX_PATH_MAX-1]=0;
+			s_unix->sun_family = AF_UNIX;
+			s_size = sizeof(*s_unix);
+			break;
+		case MODE_TCP:
+			fd_in = fd_out = socket(PF_INET, SOCK_STREAM, 0);
+			if (fd_in < 0)
+				die("git-daemon: Can't create TCP/IP socket");
+			s_inet->sin_family = AF_INET;
+			err = get_ip_addr(host, &s_inet->sin_addr);
+			s_inet->sin_port = htons(get_port(port));
+			s_size = sizeof(*s_inet);
+			break;
+	}
+
+
+	if (argc > 0) {
+		if (s_size != 0) {
+			err = connect(fd_in, &sock.generic, s_size);
+			if (err < 0)
+				die("git-daemon: Can't connect to socket \"%s\"",port);
+		}
+
+		err = send_command(argc, argv);
+		if (err < 0)
+			return err;
+		return  process_request();
+        }
+
+	if (s_size != 0) {
+		int one = 1;
+		err = setsockopt(fd_in, SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one));
+		err = bind(fd_in, &sock.generic, s_size);
+		listen(fd_in, max_clients);
+		do {
+			signal(SIGCHLD, reaper);
+			fd_out = accept(fd_in, NULL, NULL);
+			if (fd_out < 0)
+				die("git-daemon: Failed to start listening");
+			if (fork()==0) 
+				break;
+			close(fd_out);
+		} while (1);
+
+		fd_in = fd_out;
+	}
+
+	while (process_request() >= 0);
+
+	return 0;
+}
only in patch2:
unchanged:
--- /dev/null
+++ b/Documentation/git-daemon.txt
@@ -0,0 +1,92 @@
+git-daemon(1)
+=============
+v0.1, May 2005
+
+NAME
+----
+git-daemon - Client/server for exchanging GIT objects
+
+
+SYNOPSIS
+--------
+'git-daemon' [--db <git-db-path>] [--stdio | --tcp | --udp] 
+	     [--port <port>] [--host <host>] [-- command...]
+
+DESCRIPTION
+-----------
+
+OPTIONS
+-------
+--db::
+	Set the GIT object database path to use
+
+--stdio::
+	Use stdin/stdout as the transport (default)
+
+--tcp::
+	Use TCP as the transport
+
+--unix::
+	Use Unix domain sockets as the transport
+
+--host::
+	Specify IPv4 host for --tcp transport
+
+--port::
+	Specify IPv4 port for --tcp transport, or the
+	filesystem socket path for the --unix transport
+
+--::
+	Stop argument processing
+
+<command>::
+	GIT Daemon protocol command
+
+GIT Daemon Protocol Commands:
+-----------------------------
+
+version::
+	Requests the version string, returned in an error message
+	Expects: 'error'
+
+head <head-id>::
+	Requests SHA1 of the <head-id>
+	Expects: 'sha1' or 'error'
+
+head <head-id> <old-sha1> <new-sha1>::
+	Requests that <head-id> be updated with <new-sha1>. Use
+	'--' for <old-sha1> to create a new head.
+	Expects: 'sha1', 'request', or 'error'
+
+exists <sha1>::
+	Determine if SHA1 exists in remote
+	Expects: 'sha1' or 'error'
+
+request <sha1>::
+	Request SHA1 object. 
+	Expects: 'send' or 'error'
+
+send <sha1>::
+	Send SHA1 object.
+	Expects: 'sha1' or 'error'
+
+error <key> <hex-code> <error-message>::
+	Returned error code
+	Expects: Nothing
+
+sha1 <key> <hex-sha1>::
+	Success message for 'head' and 'send' commands.
+	Expects: Nothing
+
+Author
+------
+Written by Jason McMullan <jason.mcmullan@timesys.com>
+
+Documentation
+--------------
+Documentation by Jason McMullan and the git-list <git@vger.kernel.org>.
+
+GIT
+---
+Part of the link:git.html[git] suite
+
-- 
Jason McMullan <jason.mcmullan@timesys.com>
TimeSys Corporation

Re: [PATCH] git-daemon server

From: Linus Torvalds <torvalds@osdl.org>
Date: 2016-06-15 22:41:59


On Fri, 3 Jun 2005, Jason McMullan wrote:
git-daemon - Client/server for exchanging GIT objects
Hmm.. I'd prefer not calling it a deamon, since if I understand correctly,
the _normal_ usage would likely be to use it as

	.. | ssh .... -e git-deamon

ie it wouldn't be a deamon at all, it would be a foreground server. So I'd 
call it just "git-server" or something.

Anyway, technically this looks ok, but I'd hate to apply it until there is 
something that actually uses it and an example of that usage.

		Linus

Re: [PATCH] git-daemon server

From: McMullan, Jason <hidden>
Date: 2016-06-15 22:41:59

On Fri, 2005-06-03 at 09:02 -0700, Linus Torvalds wrote:
ie it wouldn't be a deamon at all, it would be a foreground server. So I'd 
call it just "git-server" or something.
Server.. Sounds ok. Will do.
Anyway, technically this looks ok, but I'd hate to apply it until there is 
something that actually uses it and an example of that usage.
Well, I use it to keep my home and work repos in sync, but yeah,
users == 1 doesn't make a good case for inclusion.

I just want to have this out there, because I find rsync a little too
heavy handed for my taste, and I'm still hesitant about the cg-rpush
locking semantics.

-- 
Jason McMullan [off-list ref]
TimeSys Corporation

Re: [PATCH] git-daemon server

From: Linus Torvalds <torvalds@osdl.org>
Date: 2016-06-15 22:41:59


On Fri, 3 Jun 2005, McMullan, Jason wrote:
quoted
Anyway, technically this looks ok, but I'd hate to apply it until there is 
something that actually uses it and an example of that usage.
Well, I use it to keep my home and work repos in sync, but yeah,
users == 1 doesn't make a good case for inclusion.
Well, it's fine, but you should at least tell _how_ you use it. What are
the scripts you use to synchronize etc etc..

So when I said "some_thing_ that actually uses it", I didn't mean more
_people_ than you personally, but the scaffolding around it, really.

			Linus

Re: [PATCH] git-daemon server

From: McMullan, Jason <hidden>
Date: 2016-06-15 22:41:59

On Fri, 2005-06-03 at 09:30 -0700, Linus Torvalds wrote:
Well, it's fine, but you should at least tell _how_ you use it. What are
the scripts you use to synchronize etc etc..
Sorry, here's some use cases (assuming that git-server is running on
192.168.1.1 port 7777)

###### Create a new local git repo

$ GITCONN="--tcp --host 192.168.1.1 --port 7777"
$ git-init-db
$ git-server $GITCONN -- request master



###### Update local repo to remote's head linux-devel

$ git-server $GITCONN -- request linux-devel


###### Create remote head linux-mumble

$ git-server $GITCONN -- head -- linux-mumble
6678abc8d819abc919a018ca89273091bc98a


###### Get remote head's linux-mumble SHA1

$ git-server $GITCONN -- head linux-mumble
sha1 linux-mumble 6678abc8d819abc919a018ca89273091bc98a


###### Set remote head to new SHA1

$ git-server $GITCONN -- head linux-mumble
6678abc8d819abc919a018ca89273091bc98a
8389324a9890218bcd9a0193781abc8a183a9
sha1 linux-mumble 8389324a9890218bcd9a0193781abc8a183a9

(This has the side effect of the server requesting SHA1 objects from the
client if they do not exist on the remote repository)


-- 
Jason McMullan [off-list ref]
TimeSys Corporation

Re: [PATCH] git-daemon server

From: Linus Torvalds <torvalds@osdl.org>
Date: 2016-06-15 22:41:59


On Fri, 3 Jun 2005, McMullan, Jason wrote:
###### Create a new local git repo

$ GITCONN="--tcp --host 192.168.1.1 --port 7777"
$ git-init-db
$ git-server $GITCONN -- request master
Ok. Am I to understand that I could do a push to a remote server with ssh 
using something like

	git-server -- send master |
		ssh master.kernel.org git-server --db /pub/scm/linux/kernel/git/torvalds/linux-2.6.git

or what?

Also, how do I send tags?

			Linus

Re: [PATCH] git-daemon server

From: McMullan, Jason <hidden>
Date: 2016-06-15 22:41:59

On Fri, 2005-06-03 at 10:41 -0700, Linus Torvalds wrote: 
Ok. Am I to understand that I could do a push to a remote server with ssh 
using something like

	git-server -- send master |
		ssh master.kernel.org git-server --db /pub/scm/linux/kernel/git/torvalds/linux-2.6.git
No, but you could do:

$ ssh master.kernel.org -L 7777:localhost:7777 \
    git-server --tcp --port 7777 \
               --db /pub/scm/linux/kernel/git/torvalds/linux-2.6.git &
$ git-server --tcp --port 7777 send master


You could probably play some games with shell redirection to file
descriptors if you really wanted to use the stdio stuff over ssh,
but that's a little ugly.

Also, how do I send tags?
Tags are on my 'TODO' list, as are the deltafication and
mulitple-request changes. Also, a verdict on the name
(daemon vs server vs anything else) would be nice.

-- 
Jason McMullan [off-list ref]
TimeSys Corporation

Re: [PATCH] git-daemon server

From: Linus Torvalds <torvalds@osdl.org>
Date: 2016-06-15 22:41:59


On Fri, 3 Jun 2005, McMullan, Jason wrote:
No, but you could do:
That doth suck. Is there any reason for not allowing stdin/stdout as the
transfer protocol?

Making games like having ssh pipe things through is what I already have to 
do with mail, and it's ok for long-running servers, but I really think of 
a "git pull" as an _event_.

So why doesn't stdin/stdout work with ssh, even though the docs say that
it's even the default?

		Linus

Re: [PATCH] git-daemon server

From: McMullan, Jason <hidden>
Date: 2016-06-15 22:41:59

Linus, you were wondering why stdin/stdout was the default, even though
I don't have a use case for ssh with stdin/stdout:

1) I used the stdin/stdout stuff as debugging

2) It works with xinetd

3) Because I can't figure out how to get /bin/sh to give me two pipes
   that hook together two processes. What I really want:

ssh user@remote git server --db /my/git.git <|> git server request HEAD

Where 'xxx <|>  yyy' means:

	Take process xxx's stdin, hook it to yyy's stdout,
	Take process yyy's stdin, hook it to xxx's stdout,
	Run till they both die.

If you know how to do that, I'd be grateful.

-- 
Jason McMullan [off-list ref]
TimeSys Corporation

Re: [PATCH] git-daemon server

From: Daniel Barkalow <hidden>
Date: 2016-06-15 22:41:59

On Fri, 3 Jun 2005, Jason McMullan wrote:
git-daemon - Client/server for exchanging GIT objects
Is this somehow different from rpush/rpull aside from using an
externally-provided socket and having the proper locking for writing refs
(which I posted a while ago, but which hasn't been included anywhere yet)?

	-Daniel
*This .sig left intentionally blank*

Re: [PATCH] git-daemon server

From: Linus Torvalds <torvalds@osdl.org>
Date: 2016-06-15 22:41:59


On Fri, 3 Jun 2005, McMullan, Jason wrote:
1) I used the stdin/stdout stuff as debugging

2) It works with xinetd

3) Because I can't figure out how to get /bin/sh to give me two pipes
   that hook together two processes. What I really want:

ssh user@remote git server --db /my/git.git <|> git server request HEAD
Ok, so a ssh connection _would_ work per se, and the only real issue is
the pipe itself is one-directional as done by the shell?
Where 'xxx <|>  yyy' means:

	Take process xxx's stdin, hook it to yyy's stdout,
	Take process yyy's stdin, hook it to xxx's stdout,
	Run till they both die.

If you know how to do that, I'd be grateful.
Yeah, you're right, you can't do bi-directional piping with shell, and 
you'd need to do it inside your program. It should be easy enough to do 
with something like adding a new flag that says "--exec", and when seeing 
that, doing something like

	if (!strcmp(argv[i], "--exec")) {
		int fd[2][2];
		pid_t pid;

		if (pipe(fd[0]) < 0 || pipe(fd[1]) < 0)
			die("unable to create pipes");
		pid = fork();
		if (pid < 0)
			die("unable to fork exec process");

		if (!pid) {
			dup2(fd[0][0], 0);
			dup2(fd[1][1], 1);
			close_pipes(fd);
			exit(system(argv[i+1]));
		}
		dup2(fd[1][0], 0);
		dup2(fd[0][1], 1);
		close_pipes(fd);
	}

where "close_pipes()" just looks like

	void close_pipes(int *fd)
	{
		int i;
		for (i = 0; i < 3; i++)
			close(fd[i]);
	}

and as usual, the above is totally and utterly untested. And using 
"system()" is cheezy and does an extra unnecessary fork(), so if you want 
to, it could be better done with just a "execve(/bin/sh -c 'string')" 
approach by hand.

Anyway, with _something_ like the above you could do something like

	git-sync --exec "ssh master.kernel.org git-sync" ....

and it would do the obvious thing.

What do you think?

		Linus

Re: [PATCH] git-daemon server

From: Linus Torvalds <torvalds@osdl.org>
Date: 2016-06-15 22:41:59


On Fri, 3 Jun 2005, Daniel Barkalow wrote:
Is this somehow different from rpush/rpull aside from using an
externally-provided socket and having the proper locking for writing refs
(which I posted a while ago, but which hasn't been included anywhere yet)?
Deathmatch! 

Anyway, I have to admit that at least as far as I'm concerned, the 
rpull/rpush thing has the same issues as Jason's code - I've not seen the 
usage documented anywhere, and that's the only real reason I don't use it 
myself.

Also, just out of interest, do either or both of these things pipeline the
transfer?

Me, I want to have a fairly simple script that does the equivalent of what
I do now for pushing:

	rsync -av --delete --exclude-from=.exclude .git/ master.kernel.org:/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

except I'd want it to do the locking and the "only accept trivial pushes" 
stuff (ie no merging, just a pure update).

Maybe git-rpush does this already, and I just never realized.

		Linus

Re: [PATCH] git-daemon server

From: Daniel Serpell <hidden>
Date: 2016-06-15 22:41:59

On 6/3/05, Linus Torvalds [off-list ref] wrote:
On Fri, 3 Jun 2005, McMullan, Jason wrote:
quoted
3) Because I can't figure out how to get /bin/sh to give me two pipes
   that hook together two processes. What I really want:

ssh user@remote git server --db /my/git.git <|> git server request HEAD
Ok, so a ssh connection _would_ work per se, and the only real issue is
the pipe itself is one-directional as done by the shell?
[...]
Anyway, with _something_ like the above you could do something like

        git-sync --exec "ssh master.kernel.org git-sync" ....

and it would do the obvious thing.
I would prefer a command line more like the rsync one:
       git-sync master.kernel.org:directory  ....

Or:
       git-sync :ssh:master.kernel.org:directory  ....

Or "uri like":
       git-sync ssh://mylogin@master.kernel.org/directory  ....

It seems much easier, and I suppose that SSH should be the
default "developer" protocol.

     Daniel.

Re: [PATCH] git-daemon server

From: Petr Baudis <hidden>
Date: 2016-06-15 22:41:59

Dear diary, on Fri, Jun 03, 2005 at 11:59:38PM CEST, I got a letter
where Linus Torvalds [off-list ref] told me that...
Anyway, I have to admit that at least as far as I'm concerned, the 
rpull/rpush thing has the same issues as Jason's code - I've not seen the 
usage documented anywhere, and that's the only real reason I don't use it 
myself.
FWIW, Cogito's cg-pull can use rpull/rpush (if your branch URLs use the
git+ssh scheme), and I've seen people actually using it.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
C++: an octopus made by nailing extra legs onto a dog. -- Steve Taylor

Re: [PATCH] git-daemon server

From: Daniel Barkalow <hidden>
Date: 2016-06-15 22:41:59

On Fri, 3 Jun 2005, Linus Torvalds wrote:

On Fri, 3 Jun 2005, Daniel Barkalow wrote:
quoted
Is this somehow different from rpush/rpull aside from using an
externally-provided socket and having the proper locking for writing refs
(which I posted a while ago, but which hasn't been included anywhere yet)?
Deathmatch! 

Anyway, I have to admit that at least as far as I'm concerned, the 
rpull/rpush thing has the same issues as Jason's code - I've not seen the 
usage documented anywhere, and that's the only real reason I don't use it 
myself.
See below. I'd foolishly explained http-pull and rpull, but neglected to
mention that rpush is actually useful in its own right. It is used in
essentially the same way, but on the other computer. (And I need to fix
the Documentation/ file, since other people seem to have missed it, too)
Also, just out of interest, do either or both of these things pipeline the
transfer?
Mine doesn't currently. The protocol supports a low level of pipelining
(you can request all the objects you know you want, and the sending side
doesn't care that you're sending requests before the previous ones are
satisfied), but I haven't figured out (or found out from Matt, more
likely) a good way to negotiate something more clever.
Me, I want to have a fairly simple script that does the equivalent of what
I do now for pushing:

	rsync -av --delete --exclude-from=.exclude .git/ master.kernel.org:/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

except I'd want it to do the locking and the "only accept trivial pushes" 
stuff (ie no merging, just a pure update).
With patches I have (but have to rebase and such), you could do:

git-rpush -a -w heads/master heads/master //master.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

This would only send the heads/master file and all of the object files it
references. It would be easy to write a "--old heads/linus" which would
require that the remote value was the local value of heads/linus (I.e.,
the public value of your head that you'd been working from).
Maybe git-rpush does this already, and I just never realized.
It currently does all the objects, but doesn't write the refs file on the
remote end. That is:

  git-rpush -a heads/master //master.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

Will cause the public tree to contain all the objects reachable from the
local value of heads/master. Then you need to write the new head by hand
if you don't have the newer stuff.

	-Daniel
*This .sig left intentionally blank*

Re: [PATCH] git-daemon server

From: Jason McMullan <hidden>
Date: 2016-06-15 22:41:59

On Fri, 2005-06-03 at 14:38 -0700, Linus Torvalds wrote:
Anyway, with _something_ like the above you could do something like

	git-sync --exec "ssh master.kernel.org git-sync" ....

and it would do the obvious thing.
Sounds good. I'll implement that right after I put in tags support, and
merge by verify-before-write pull mechanism into pull.c, so everyone
can use it.

And, it looks like 'git-sync' is the decided name? Excellent!

Re: [PATCH] git-daemon server

From: Jason McMullan <hidden>
Date: 2016-06-15 22:41:59

On Fri, 2005-06-03 at 14:59 -0700, Linus Torvalds wrote:
Me, I want to have a fairly simple script that does the equivalent of what
I do now for pushing:

	rsync -av --delete --exclude-from=.exclude .git/ master.kernel.org:/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

except I'd want it to do the locking and the "only accept trivial pushes" 
stuff (ie no merging, just a pure update).

$ GITCONNECTION="--tcp ...."

$ git-sync $GITCONNECTION head master master `cat .git/HEAD`

(Yeah, the 'master master' is a cheezy way to do an update, but
 if you're the only one who can update the head, it's a non-issue)

Re: [PATCH] git-daemon server

From: Linus Torvalds <torvalds@osdl.org>
Date: 2016-06-15 22:41:59


On Sun, 5 Jun 2005, Jason McMullan wrote:
On Fri, 2005-06-03 at 14:59 -0700, Linus Torvalds wrote:
quoted
Me, I want to have a fairly simple script that does the equivalent of what
I do now for pushing:

	rsync -av --delete --exclude-from=.exclude .git/ master.kernel.org:/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

except I'd want it to do the locking and the "only accept trivial pushes" 
stuff (ie no merging, just a pure update).

$ GITCONNECTION="--tcp ...."
Nope. I'm not going to run untrusted deamons of my own on master.

I'm serious about the "ssh or nothing" thing. Anything but a direct ssh 
conection is simply not an option.

		Linus
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help