Hi,
I've decided to get this series merged now instead of waiting for the
ternary treap refactor. It's in excellent shape, thanks to Jonathan's
constant reviews/ fixes and David's constant refactoring. Since
Jonathan's last send, I've incoporated some suggestions from my own
review of Jonathan's series and split it into two series as I proposed
earlier; I'll send the second series (rr/contrib-svn-fe) shortly after
this.
Once this series is merged, I estimate that the following will come in
as incremental patches when the work is finished:
rr/ternary-trp-refactor
rr/zero-tree-refactor
rr/dumpfilev3-parser
Thanks.
-- Ram
David Barr (5):
Add memory pool library
Add string-specific memory pool
Add stream helper library
Add infrastructure to write revisions in fast-export format
Add SVN dump parser
Jason Evans (1):
Add treap implementation
Jonathan Nieder (2):
Export parse_date_basic() to convert a date string to timestamp
Introduce vcs-svn lib
Makefile | 12 ++-
cache.h | 1 +
date.c | 14 +--
vcs-svn/LICENSE | 33 +++++
vcs-svn/fast_export.c | 75 +++++++++++
vcs-svn/fast_export.h | 14 ++
vcs-svn/line_buffer.c | 93 ++++++++++++++
vcs-svn/line_buffer.h | 14 ++
vcs-svn/obj_pool.h | 80 ++++++++++++
vcs-svn/repo_tree.c | 335 +++++++++++++++++++++++++++++++++++++++++++++++++
vcs-svn/repo_tree.h | 26 ++++
vcs-svn/string_pool.c | 114 +++++++++++++++++
vcs-svn/string_pool.h | 15 +++
vcs-svn/svndump.c | 289 ++++++++++++++++++++++++++++++++++++++++++
vcs-svn/svndump.h | 8 ++
vcs-svn/trp.h | 220 ++++++++++++++++++++++++++++++++
vcs-svn/trp.txt | 102 +++++++++++++++
17 files changed, 1436 insertions(+), 9 deletions(-)
create mode 100644 vcs-svn/LICENSE
create mode 100644 vcs-svn/fast_export.c
create mode 100644 vcs-svn/fast_export.h
create mode 100644 vcs-svn/line_buffer.c
create mode 100644 vcs-svn/line_buffer.h
create mode 100644 vcs-svn/obj_pool.h
create mode 100644 vcs-svn/repo_tree.c
create mode 100644 vcs-svn/repo_tree.h
create mode 100644 vcs-svn/string_pool.c
create mode 100644 vcs-svn/string_pool.h
create mode 100644 vcs-svn/svndump.c
create mode 100644 vcs-svn/svndump.h
create mode 100644 vcs-svn/trp.h
create mode 100644 vcs-svn/trp.txt
From: Jonathan Nieder <redacted>
approxidate() is not appropriate for reading machine-written dates
because it guesses instead of erroring out on malformed dates.
parse_date() is less convenient since it returns its output as a
string. So export the underlying function that writes a timestamp.
While at it, change the return value to match the usual convention:
return 0 for success and -1 for failure.
Signed-off-by: Jonathan Nieder <redacted>
Acked-by: Ramkumar Ramachandra <redacted>
Signed-off-by: Ramkumar Ramachandra <redacted>
---
cache.h | 1 +
date.c | 14 ++++++--------
2 files changed, 7 insertions(+), 8 deletions(-)
@@ -811,6 +811,7 @@ const char *show_date_relative(unsigned long time, int tz,char*timebuf,size_ttimebuf_size);intparse_date(constchar*date,char*buf,intbufsize);+intparse_date_basic(constchar*date,unsignedlong*timestamp,int*offset);voiddatestamp(char*buf,intbufsize);#define approxidate(s) approxidate_careful((s), NULL)unsignedlongapproxidate_careful(constchar*,int*);
@@ -586,7 +586,7 @@ static int date_string(unsigned long date, int offset, char *buf, int len)/* Gr. strptime is crap for this; it doesn't have a way to require RFC2822(i.e.English)day/monthnames,anditdoesn'tworkcorrectlywith%z.*/-intparse_date_toffset(constchar*date,unsignedlong*timestamp,int*offset)+intparse_date_basic(constchar*date,unsignedlong*timestamp,int*offset){structtmtm;inttm_gmt;
@@ -642,17 +642,16 @@ int parse_date_toffset(const char *date, unsigned long *timestamp, int *offset)if(!tm_gmt)*timestamp-=*offset*60;-return1;/* success */+return0;/* success */}intparse_date(constchar*date,char*result,intmaxlen){unsignedlongtimestamp;intoffset;-if(parse_date_toffset(date,×tamp,&offset)>0)-returndate_string(timestamp,offset,result,maxlen);-else+if(parse_date_basic(date,×tamp,&offset))return-1;+returndate_string(timestamp,offset,result,maxlen);}enumdate_modeparse_date_format(constchar*format)
From: David Barr <redacted>
Intern strings so they can be compared by address and stored without
wasting space.
This library uses the macros in the obj_pool.h and trp.h to create a
memory pool for strings and expose an API for handling them.
Signed-off-by: David Barr <redacted>
Signed-off-by: Ramkumar Ramachandra <redacted>
Signed-off-by: Junio C Hamano <redacted>
Signed-off-by: Jonathan Nieder <redacted>
---
Makefile | 4 +-
vcs-svn/string_pool.c | 114 +++++++++++++++++++++++++++++++++++++++++++++++++
vcs-svn/string_pool.h | 15 ++++++
3 files changed, 131 insertions(+), 2 deletions(-)
create mode 100644 vcs-svn/string_pool.c
create mode 100644 vcs-svn/string_pool.h
@@ -0,0 +1,114 @@+/*+*Licensedunderatwo-clauseBSD-stylelicense.+*SeeLICENSEfordetails.+*/++#include"git-compat-util.h"+#include"trp.h"+#include"obj_pool.h"+#include"string_pool.h"++staticstructtrp_roottree={~0};++structnode{+uint32_toffset;+structtrp_nodechildren;+};++/* Two memory pools: one for struct node, and another for strings */+obj_pool_gen(node,structnode,4096);+obj_pool_gen(string,char,4096);++staticchar*node_value(structnode*node)+{+returnnode?string_pointer(node->offset):NULL;+}++staticintnode_cmp(structnode*a,structnode*b)+{+returnstrcmp(node_value(a),node_value(b));+}++/* Build a Treap from the node structure (a trp_node w/ offset) */+trp_gen(static,tree_,structnode,children,node,node_cmp);++char*pool_fetch(uint32_tentry)+{+returnnode_value(node_pointer(entry));+}++uint32_tpool_intern(char*key)+{+/* Canonicalize key */+structnode*match=NULL;+uint32_tkey_len;+if(key==NULL)+return~0;+key_len=strlen(key)+1;+structnode*node=node_pointer(node_alloc(1));+node->offset=string_alloc(key_len);+strcpy(node_value(node),key);+match=tree_search(&tree,node);+if(!match){+tree_insert(&tree,node);+}else{+node_free(1);+string_free(key_len);+node=match;+}+returnnode_offset(node);+}++uint32_tpool_tok_r(char*str,constchar*delim,char**saveptr)+{+char*token=strtok_r(str,delim,saveptr);+returntoken?pool_intern(token):~0;+}++voidpool_print_seq(uint32_tlen,uint32_t*seq,chardelim,FILE*stream)+{+uint32_ti;+for(i=0;i<len&&~seq[i];i++){+fputs(pool_fetch(seq[i]),stream);+if(i<len-1&&~seq[i+1])+fputc(delim,stream);+}+}++uint32_tpool_tok_seq(uint32_tmax,uint32_t*seq,char*delim,char*str)+{+char*context=NULL;+uint32_tlength=0,token=str?pool_tok_r(str,delim,&context):~0;+while(length<max){+seq[length++]=token;+if(token==~0)+break;+token=pool_tok_r(NULL,delim,&context);+}+seq[length?length-1:0]=~0;+returnlength;+}++voidpool_init(void)+{+uint32_tnode;+uint32_tstring=0;+string_init();+while(string<string_pool.size){+node=node_alloc(1);+node_pointer(node)->offset=string;+tree_insert(&tree,node_pointer(node));+string+=strlen(string_pointer(string))+1;+}+}++voidpool_commit(void)+{+string_commit();+}++voidpool_reset(void)+{+node_reset();+string_reset();+}
From: David Barr <redacted>
Add a memory pool library implemented using C macros. The obj_pool_gen()
macro creates a type-specific memory pool API.
The memory pool library is distinguished from the existing specialized
allocators in alloc.c by using a contiguous block for all allocations.
This means that on one hand, long-lived pointers have to be written as
offsets, since the base address changes as the pool grows, but on the
other hand, the entire pool can be easily written to the file system.
This allows the memory pool to persist between runs of an application.
For the svn importer, such a facility is useful because each svn
revision can copy trees and files from any previous revision. The
relevant information for all revisions has to persist somehow to
support incremental runs, and for now it is simplest to avoid relying
on the target VCS for that.
obj_pool_gen(pre, obj_t, initial_capability)
pre: Prefix for generated functions (example: string).
obj_t: Type for treap data structure (example: char).
initial_capacity: Initial size of the memory pool (example: 4096).
void pre_init(void);
Read values from a previous run to initialize the pool.
If this function is not called, the pool begins valid but empty.
uint32_t pre_alloc(uint32_t nmemb);
Reserve space for a few objects in the pool and return an
offset to the first one.
uint32_t pre_free(uint32_t nmemb);
Unreserve the last few objects reserved.
uint32_t pre_offset(obj_t *pointer);
obj_t *pre_pointer(uint32_t offset);
Convert between pointers into the in-memory pool and offsets
from the beginning (or ~0 for the NULL pointer). Pointers are
not guaranteed to remain valid after a pre_alloc() operation
or pre_reset() followed by pre_init(), but offsets are.
void pre_commit(void);
Write the pool to file. A pre_reset() followed by pre_init()
(pehaps with exit() in between) will return the pool to the
last committed state.
void pre_reset(void);
Deinitialize the pool, freeing any associated memory and
file handles.
Signed-off-by: David Barr <redacted>
Signed-off-by: Ramkumar Ramachandra <redacted>
Signed-off-by: Junio C Hamano <redacted>
Signed-off-by: Jonathan Nieder <redacted>
---
Makefile | 3 +-
vcs-svn/LICENSE | 26 +++++++++++++++++
vcs-svn/obj_pool.h | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 108 insertions(+), 1 deletions(-)
create mode 100644 vcs-svn/LICENSE
create mode 100644 vcs-svn/obj_pool.h
@@ -0,0 +1,26 @@+Copyright (C) 2010 David Barr <david.barr@cordelta.com>.+All rights reserved.++Redistribution and use in source and binary forms, with or without+modification, are permitted provided that the following conditions+are met:+1. Redistributions of source code must retain the above copyright+ notice(s), this list of conditions and the following disclaimer+ unmodified other than the allowable addition of one or more+ copyright notices.+2. Redistributions in binary form must reproduce the above copyright+ notice(s), this list of conditions and the following disclaimer in+ the documentation and/or other materials provided with the+ distribution.++THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY+EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR+PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER(S) BE+LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR+BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,+WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE+OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,+EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
From: David Barr <redacted>
This library provides thread-unsafe fgets()- and fread()-like
functions where the caller does not have to supply a buffer. It
maintains a couple of static buffers and provides an API to use
them.
NEEDSWORK: what should buffer_copy_bytes do on error?
Signed-off-by: David Barr <redacted>
Signed-off-by: Ramkumar Ramachandra <redacted>
Signed-off-by: Junio C Hamano <redacted>
Signed-off-by: Jonathan Nieder <redacted>
---
Makefile | 5 ++-
vcs-svn/line_buffer.c | 93 +++++++++++++++++++++++++++++++++++++++++++++++++
vcs-svn/line_buffer.h | 14 +++++++
3 files changed, 110 insertions(+), 2 deletions(-)
create mode 100644 vcs-svn/line_buffer.c
create mode 100644 vcs-svn/line_buffer.h
@@ -0,0 +1,93 @@+/*+*Licensedunderatwo-clauseBSD-stylelicense.+*SeeLICENSEfordetails.+*/++#include"git-compat-util.h"++#include"line_buffer.h"+#include"obj_pool.h"++#define LINE_BUFFER_LEN 10000+#define COPY_BUFFER_LEN 4096++/* Create memory pool for char sequence of known length */+obj_pool_gen(blob,char,4096);++staticcharline_buffer[LINE_BUFFER_LEN];+staticcharbyte_buffer[COPY_BUFFER_LEN];+staticFILE*infile;++intbuffer_init(constchar*filename)+{+infile=filename?fopen(filename,"r"):stdin;+if(!infile)+return-1;+return0;+}++intbuffer_deinit()+{+fclose(infile);+return0;+}++/* Read a line without trailing newline. */+char*buffer_read_line(void)+{+char*end;+if(!fgets(line_buffer,sizeof(line_buffer),infile))+/* Error or data exhausted. */+returnNULL;+end=line_buffer+strlen(line_buffer);+if(end[-1]=='\n')+end[-1]='\0';+elseif(feof(infile))+;/* No newline at end of file. That's fine. */+else+/*+*Linewastoolong.+*Thereisprobablyasanerwaytodealwiththis,+*butfornowlet'sreturnanerror.+*/+returnNULL;+returnline_buffer;+}++char*buffer_read_string(uint32_tlen)+{+char*s;+blob_free(blob_pool.size);+s=blob_pointer(blob_alloc(len+1));+s[fread(s,1,len,infile)]='\0';+returnferror(infile)?NULL:s;+}++voidbuffer_copy_bytes(uint32_tlen)+{+uint32_tin;+while(len>0&&!feof(infile)){+in=len<COPY_BUFFER_LEN?len:COPY_BUFFER_LEN;+in=fread(byte_buffer,1,in,infile);+len-=in;+fwrite(byte_buffer,1,in,stdout);+if(ferror(infile)||ferror(stdout))+/* NEEDSWORK: handle error. */+break;+}+}++voidbuffer_skip_bytes(uint32_tlen)+{+uint32_tin;+while(len>0&&!feof(infile)&&!ferror(infile)){+in=len<COPY_BUFFER_LEN?len:COPY_BUFFER_LEN;+in=fread(byte_buffer,1,in,infile);+len-=in;+}+}++voidbuffer_reset(void)+{+blob_reset();+}
From: Jonathan Nieder <redacted>
Teach the build system to build a separate library for the
upcoming subversion interop support.
The resulting vcs-svn/lib.a does not contain any code, nor is
it built during a normal build. This is just scaffolding for
later changes.
Signed-off-by: Jonathan Nieder <redacted>
Acked-by: Ramkumar Ramachandra <redacted>
Signed-off-by: Ramkumar Ramachandra <redacted>
---
Makefile | 8 +++++++-
1 files changed, 7 insertions(+), 1 deletions(-)
From: David Barr <redacted>
svndump parses data that is in SVN dumpfile format produced by
`svnadmin dump` with the help of line_buffer and uses repo_tree and
fast_export to emit a git fast-import stream.
Based roughly on com.hydrografix.svndump 0.92 from the SvnToCCase
project at <http://svn2cc.sarovar.org/>, by Stefan Hegny and
others.
Signed-off-by: David Barr <redacted>
Signed-off-by: Ramkumar Ramachandra <redacted>
Signed-off-by: Junio C Hamano <redacted>
Signed-off-by: Jonathan Nieder <redacted>
---
Makefile | 5 +-
vcs-svn/LICENSE | 4 +
vcs-svn/svndump.c | 289 +++++++++++++++++++++++++++++++++++++++++++++++++++++
vcs-svn/svndump.h | 8 ++
4 files changed, 304 insertions(+), 2 deletions(-)
create mode 100644 vcs-svn/svndump.c
create mode 100644 vcs-svn/svndump.h
@@ -4,6 +4,10 @@ All rights reserved. Copyright (C) 2008 Jason Evans <jasone@canonware.com>. All rights reserved.+Copyright (C) 2005 Stefan Hegny, hydrografix Consulting GmbH,+Frankfurt/Main, Germany+and others, see http://svn2cc.sarovar.org+ Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
From: Jason Evans <redacted>
Provide macros to generate a type-specific treap implementation and
various functions to operate on it. It uses obj_pool.h to store memory
nodes in a treap. Previously committed nodes are never removed from
the pool; after any *_commit operation, it is assumed (correctly, in
the case of svn-fast-export) that someone else must care about them.
Treaps provide a memory-efficient binary search tree structure.
Insertion/deletion/search are about as about as fast in the average
case as red-black trees and the chances of worst-case behavior are
vanishingly small, thanks to (pseudo-)randomness. The bad worst-case
behavior is a small price to pay, given that treaps are much simpler
to implement.
From http://www.canonware.com/download/trp/trp_hash/trp.h
[db: Altered to reference nodes by offset from a common base pointer]
[db: Bob Jenkins' hashing implementation dropped for Knuth's]
[db: Methods unnecessary for search and insert dropped]
Signed-off-by: David Barr <redacted>
Signed-off-by: Ramkumar Ramachandra <redacted>
Signed-off-by: Junio C Hamano <redacted>
Signed-off-by: Jonathan Nieder <redacted>
---
Makefile | 2 +-
vcs-svn/LICENSE | 3 +
vcs-svn/trp.h | 220 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
vcs-svn/trp.txt | 102 +++++++++++++++++++++++++
4 files changed, 326 insertions(+), 1 deletions(-)
create mode 100644 vcs-svn/trp.h
create mode 100644 vcs-svn/trp.txt
@@ -1,6 +1,9 @@ Copyright (C) 2010 David Barr <david.barr@cordelta.com>. All rights reserved.+Copyright (C) 2008 Jason Evans <jasone@canonware.com>.+All rights reserved.+ Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
@@ -0,0 +1,102 @@+Motivation+==========++Treaps provide a memory-efficient binary search tree structure.+Insertion/deletion/search are about as about as fast in the average+case as red-black trees and the chances of worst-case behavior are+vanishingly small, thanks to (pseudo-)randomness. The bad worst-case+behavior is a small price to pay, given that treaps are much simpler+to implement.++From http://www.canonware.com/download/trp/trp_hash/trp.h++API+===++The trp API generates a data structure and functions to handle a+large growing set of objects stored in a pool.++The caller:++. Specifies parameters for the generated functions with the+ trp_gen(static, foo_, ...) macro.++. Allocates and clears a `struct trp_node` variable.++. Adds new items to the set using `foo_insert`.++. Can find a specific item in the set using `foo_search`.++. Can iterate over items in the set using `foo_first` and `foo_next`.++. Can remove an item from the set using `foo_remove`.++. The set is never freed.++Example:++----+struct ex_node {+ const char *s;+ struct trp_node ex_link;+};+static struct trp_root ex_base;+obj_pool_gen(ex, struct ex_node, 4096);+trp_gen(static, ex_, struct ex_node, ex_link, ex, strcmp)+struct ex_node *item;++item = ex_pointer(ex_alloc(1));+item->s = "hello";+ex_insert(&ex_base, item);+item = ex_pointer(ex_alloc(1));+item->s = "goodbye";+ex_insert(&ex_base, item);+for (item = ex_first(&ex_base); item; item = ex_next(&ex_base, item))+ printf("%s\n", item->s);+----++Functions+---------++trp_gen(attr, foo_, node_type, link_field, pool, cmp)::++ Generate a type-specific treap implementation.+++. The storage class for generated functions will be 'attr' (e.g., `static`).+. Generated function names are prefixed with 'foo_' (e.g., `treap_`).+. Treap nodes will be of type 'node_type' (e.g., `struct treap_node`).+ This type must be a struct with at least one `struct trp_node` field+ to point to its children.+. The field used to access child nodes will be 'link_field'.+. All treap nodes must lie in the 'pool' object pool.+. Treap nodes must be totally ordered by the 'cmp' relation, with the+ following prototype:+++int (*cmp)(node_type \*a, node_type \*b)+++and returning a value less than, equal to, or greater than zero+according to the result of comparison.++void foo_insert(struct trp_root *treap, node_type \*node)::++ Insert node into treap. If inserted multiple times,+ a node will appear in the treap multiple times.++void foo_remove(struct trp_root *treap, node_type \*node)::++ Remove node from treap. Caller must ensure node is+ present in treap before using this function.++node_type *foo_search(struct trp_root \*treap, node_type \*key)::++ Search for a node that matches key. If no match is found,+ return what would be key's successor, were key in treap+ (NULL if no successor).++node_type *foo_first(struct trp_root \*treap)::++ Find the first item from the treap, in sorted order.++node_type *foo_next(struct trp_root \*treap, node_type \*node)::++ Find the next item.
From: David Barr <redacted>
repo_tree maintains the exporter's state and provides a facility to to
call fast_export, which writes objects to stdout suitable for
consumption by fast-import.
The exported functions roughly correspond to Subversion FS operations.
. repo_add, repo_modify, repo_copy, repo_replace, and repo_delete
update the current commit, based roughly on the corresponding
Subversion FS operation.
. repo_commit calls out to fast_export to write the current commit to
the fast-import stream in stdout.
. repo_diff is used by the fast_export module to write the changes
for a commit.
. repo_reset erases the exporter's state, so valgrind can be happy.
Signed-off-by: David Barr <redacted>
Signed-off-by: Ramkumar Ramachandra <redacted>
Signed-off-by: Junio C Hamano <redacted>
Signed-off-by: Jonathan Nieder <redacted>
---
Makefile | 5 +-
vcs-svn/fast_export.c | 75 +++++++++++
vcs-svn/fast_export.h | 14 ++
vcs-svn/repo_tree.c | 335 +++++++++++++++++++++++++++++++++++++++++++++++++
vcs-svn/repo_tree.h | 26 ++++
5 files changed, 453 insertions(+), 2 deletions(-)
create mode 100644 vcs-svn/fast_export.c
create mode 100644 vcs-svn/fast_export.h
create mode 100644 vcs-svn/repo_tree.c
create mode 100644 vcs-svn/repo_tree.h
@@ -0,0 +1,335 @@+/*+*Licensedunderatwo-clauseBSD-stylelicense.+*SeeLICENSEfordetails.+*/++#include"git-compat-util.h"++#include"string_pool.h"+#include"repo_tree.h"+#include"obj_pool.h"+#include"fast_export.h"++#include"trp.h"++structrepo_dirent{+uint32_tname_offset;+structtrp_nodechildren;+uint32_tmode;+uint32_tcontent_offset;+};++structrepo_dir{+structtrp_rootentries;+};++structrepo_commit{+uint32_troot_dir_offset;+};++/* Memory pools for commit, dir and dirent */+obj_pool_gen(commit,structrepo_commit,4096);+obj_pool_gen(dir,structrepo_dir,4096);+obj_pool_gen(dirent,structrepo_dirent,4096);++staticuint32_tactive_commit;+staticuint32_tmark;++staticintrepo_dirent_name_cmp(constvoid*a,constvoid*b);++/* Treap for directory entries */+trp_gen(static,dirent_,structrepo_dirent,children,dirent,repo_dirent_name_cmp);++uint32_tnext_blob_mark(void)+{+returnmark++;+}++staticstructrepo_dir*repo_commit_root_dir(structrepo_commit*commit)+{+returndir_pointer(commit->root_dir_offset);+}++staticstructrepo_dirent*repo_first_dirent(structrepo_dir*dir)+{+returndirent_first(&dir->entries);+}++staticintrepo_dirent_name_cmp(constvoid*a,constvoid*b)+{+conststructrepo_dirent*dirent1=a,*dirent2=b;+uint32_ta_offset=dirent1->name_offset;+uint32_tb_offset=dirent2->name_offset;+return(a_offset>b_offset)-(a_offset<b_offset);+}++staticintrepo_dirent_is_dir(structrepo_dirent*dirent)+{+returndirent!=NULL&&dirent->mode==REPO_MODE_DIR;+}++staticstructrepo_dir*repo_dir_from_dirent(structrepo_dirent*dirent)+{+if(!repo_dirent_is_dir(dirent))+returnNULL;+returndir_pointer(dirent->content_offset);+}++staticstructrepo_dir*repo_clone_dir(structrepo_dir*orig_dir)+{+uint32_torig_o,new_o;+orig_o=dir_offset(orig_dir);+if(orig_o>=dir_pool.committed)+returnorig_dir;+new_o=dir_alloc(1);+orig_dir=dir_pointer(orig_o);+*dir_pointer(new_o)=*orig_dir;+returndir_pointer(new_o);+}++staticstructrepo_dirent*repo_read_dirent(uint32_trevision,uint32_t*path)+{+uint32_tname=0;+structrepo_dirent*key=dirent_pointer(dirent_alloc(1));+structrepo_dir*dir=NULL;+structrepo_dirent*dirent=NULL;+dir=repo_commit_root_dir(commit_pointer(revision));+while(~(name=*path++)){+key->name_offset=name;+dirent=dirent_search(&dir->entries,key);+if(dirent==NULL||!repo_dirent_is_dir(dirent))+break;+dir=repo_dir_from_dirent(dirent);+}+dirent_free(1);+returndirent;+}++staticvoidrepo_write_dirent(uint32_t*path,uint32_tmode,+uint32_tcontent_offset,uint32_tdel)+{+uint32_tname,revision,dir_o=~0,parent_dir_o=~0;+structrepo_dir*dir;+structrepo_dirent*key;+structrepo_dirent*dirent=NULL;+revision=active_commit;+dir=repo_commit_root_dir(commit_pointer(revision));+dir=repo_clone_dir(dir);+commit_pointer(revision)->root_dir_offset=dir_offset(dir);+while(~(name=*path++)){+parent_dir_o=dir_offset(dir);++key=dirent_pointer(dirent_alloc(1));+key->name_offset=name;++dirent=dirent_search(&dir->entries,key);+if(dirent==NULL)+dirent=key;+else+dirent_free(1);++if(dirent==key){+dirent->mode=REPO_MODE_DIR;+dirent->content_offset=0;+dirent_insert(&dir->entries,dirent);+}++if(dirent_offset(dirent)<dirent_pool.committed){+dir_o=repo_dirent_is_dir(dirent)?+dirent->content_offset:~0;+dirent_remove(&dir->entries,dirent);+dirent=dirent_pointer(dirent_alloc(1));+dirent->name_offset=name;+dirent->mode=REPO_MODE_DIR;+dirent->content_offset=dir_o;+dirent_insert(&dir->entries,dirent);+}++dir=repo_dir_from_dirent(dirent);+dir=repo_clone_dir(dir);+dirent->content_offset=dir_offset(dir);+}+if(dirent==NULL)+return;+dirent->mode=mode;+dirent->content_offset=content_offset;+if(del&&~parent_dir_o)+dirent_remove(&dir_pointer(parent_dir_o)->entries,dirent);+}++uint32_trepo_copy(uint32_trevision,uint32_t*src,uint32_t*dst)+{+uint32_tmode=0,content_offset=0;+structrepo_dirent*src_dirent;+src_dirent=repo_read_dirent(revision,src);+if(src_dirent!=NULL){+mode=src_dirent->mode;+content_offset=src_dirent->content_offset;+repo_write_dirent(dst,mode,content_offset,0);+}+returnmode;+}++voidrepo_add(uint32_t*path,uint32_tmode,uint32_tblob_mark)+{+repo_write_dirent(path,mode,blob_mark,0);+}++uint32_trepo_replace(uint32_t*path,uint32_tblob_mark)+{+uint32_tmode=0;+structrepo_dirent*src_dirent;+src_dirent=repo_read_dirent(active_commit,path);+if(src_dirent!=NULL){+mode=src_dirent->mode;+repo_write_dirent(path,mode,blob_mark,0);+}+returnmode;+}++voidrepo_modify(uint32_t*path,uint32_tmode,uint32_tblob_mark)+{+structrepo_dirent*src_dirent;+src_dirent=repo_read_dirent(active_commit,path);+if(src_dirent!=NULL&&blob_mark==0)+blob_mark=src_dirent->content_offset;+repo_write_dirent(path,mode,blob_mark,0);+}++voidrepo_delete(uint32_t*path)+{+repo_write_dirent(path,0,0,1);+}++staticvoidrepo_git_add_r(uint32_tdepth,uint32_t*path,structrepo_dir*dir);++staticvoidrepo_git_add(uint32_tdepth,uint32_t*path,structrepo_dirent*dirent)+{+if(repo_dirent_is_dir(dirent))+repo_git_add_r(depth,path,repo_dir_from_dirent(dirent));+else+fast_export_modify(depth,path,+dirent->mode,dirent->content_offset);+}++staticvoidrepo_git_add_r(uint32_tdepth,uint32_t*path,structrepo_dir*dir)+{+structrepo_dirent*de=repo_first_dirent(dir);+while(de){+path[depth]=de->name_offset;+repo_git_add(depth+1,path,de);+de=dirent_next(&dir->entries,de);+}+}++staticvoidrepo_diff_r(uint32_tdepth,uint32_t*path,structrepo_dir*dir1,+structrepo_dir*dir2)+{+structrepo_dirent*de1,*de2;+de1=repo_first_dirent(dir1);+de2=repo_first_dirent(dir2);++while(de1&&de2){+if(de1->name_offset<de2->name_offset){+path[depth]=de1->name_offset;+fast_export_delete(depth+1,path);+de1=dirent_next(&dir1->entries,de1);+continue;+}+if(de1->name_offset>de2->name_offset){+path[depth]=de2->name_offset;+repo_git_add(depth+1,path,de2);+de2=dirent_next(&dir2->entries,de2);+continue;+}+path[depth]=de1->name_offset;++if(de1->mode==de2->mode&&+de1->content_offset==de2->content_offset){+;/* No change. */+}elseif(repo_dirent_is_dir(de1)&&repo_dirent_is_dir(de2)){+repo_diff_r(depth+1,path,+repo_dir_from_dirent(de1),+repo_dir_from_dirent(de2));+}elseif(!repo_dirent_is_dir(de1)&&!repo_dirent_is_dir(de2)){+repo_git_add(depth+1,path,de2);+}else{+fast_export_delete(depth+1,path);+repo_git_add(depth+1,path,de2);+}+de1=dirent_next(&dir1->entries,de1);+de2=dirent_next(&dir2->entries,de2);+}+while(de1){+path[depth]=de1->name_offset;+fast_export_delete(depth+1,path);+de1=dirent_next(&dir1->entries,de1);+}+while(de2){+path[depth]=de2->name_offset;+repo_git_add(depth+1,path,de2);+de2=dirent_next(&dir2->entries,de2);+}+}++staticuint32_tpath_stack[REPO_MAX_PATH_DEPTH];++voidrepo_diff(uint32_tr1,uint32_tr2)+{+repo_diff_r(0,+path_stack,+repo_commit_root_dir(commit_pointer(r1)),+repo_commit_root_dir(commit_pointer(r2)));+}++voidrepo_commit(uint32_trevision,uint32_tauthor,char*log,uint32_tuuid,+uint32_turl,unsignedlongtimestamp)+{+fast_export_commit(revision,author,log,uuid,url,timestamp);+pool_commit();+dirent_commit();+dir_commit();+commit_commit();+active_commit=commit_alloc(1);+commit_pointer(active_commit)->root_dir_offset=+commit_pointer(active_commit-1)->root_dir_offset;+}++staticvoidmark_init(void)+{+uint32_ti;+mark=0;+for(i=0;i<dirent_pool.size;i++)+if(!repo_dirent_is_dir(dirent_pointer(i))&&+dirent_pointer(i)->content_offset>mark)+mark=dirent_pointer(i)->content_offset;+mark++;+}++voidrepo_init(){+pool_init();+commit_init();+dir_init();+dirent_init();+mark_init();+if(commit_pool.size==0){+/* Create empty tree for commit 0. */+commit_alloc(1);+commit_pointer(0)->root_dir_offset=dir_alloc(1);+dir_pointer(0)->entries.trp_root=~0;+dir_commit();+commit_commit();+}+/* Preallocate next commit, ready for changes. */+active_commit=commit_alloc(1);+commit_pointer(active_commit)->root_dir_offset=+commit_pointer(active_commit-1)->root_dir_offset;+}++voidrepo_reset(void)+{+pool_reset();+commit_reset();+dir_reset();+dirent_reset();+}
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:08
Ramkumar Ramachandra wrote:
approxidate() is not appropriate for reading machine-written dates
because it guesses instead of erroring out on malformed dates.
parse_date() is less convenient since it returns its output as a
string. So export the underlying function that writes a timestamp.
While at it, change the return value to match the usual convention:
return 0 for success and -1 for failure.
Junio: I think this should be ejected from the series as an
independently useful cleanup.
Currently parse_date_toffset() is exported but not declared anywhere.
This patch gives it a more predictable API and adds a declaration.
Ram: thanks for the reminder.
$ make vcs-svn/lib.a V=1
rm -f vcs-svn/lib.a && ar rcs vcs-svn/lib.a
ar: vcs-svn/lib.a: No such file or directory
make: *** [vcs-svn/lib.a] Error 1
That is because the vcs-svn directory does not exist. So
probably the LICENSE should be added with the same patch
(and git should learn to track empty directories).
Jonathan
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:08
Ramkumar Ramachandra wrote:
void pre_commit(void);
Write the pool to file.
Except as a proof of concept, this is the wrong API to have. The problem
is that the caller cannot choose the filename, so it ends up being a .bin
file in the current directory, wherever that is.
The log message leaves out a subtlety: this also increases the
‘committed’ value, and bookkeeping for that might be useful to some
callers.
In other words:
If you just want something working, I’d suggest stubbing this out:
static MAYBE_UNUSED void pre##_init(void) \
{ \
} \
It even almost makes sense as API: the _init function does all
initialization tasks required, which is to say, none. (The {0, ...}
initializer already has taken care of setting all fields to 0).
This can be simplified
static MAYBE_UNUSED void pre##_commit(void) \
{ \
pre##_pool.committed = pre##_pool.size; \
} \
In other words, maybe something like this on top? This includes the
vestigal _init() function which really should not be there (it is
confusing that some callers use it and others don’t). I did not
spend much time on it because in the end I suspect we might throw
obj_pool away anyway.
---
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:08
Ramkumar Ramachandra wrote:
Treaps provide a memory-efficient binary search tree structure.
Insertion/deletion/search are about as about as fast in the average
case as red-black trees and the chances of worst-case behavior are
vanishingly small, thanks to (pseudo-)randomness. The bad worst-case
behavior is a small price to pay, given that treaps are much simpler
to implement.
I still haven’t checked this implementation in detail, but it seemed
to work in practice and is about to change anyway.
I like the documentation updates. What else changed from the
previous round?
Except as a proof of concept, this is the wrong API to have. The problem
is that the caller cannot choose the filename, so it ends up being a .bin
file in the current directory, wherever that is.
The log message leaves out a subtlety: this also increases the
‘committed’ value, and bookkeeping for that might be useful to some
callers.
In other words:
If you just want something working, I’d suggest stubbing this out:
static MAYBE_UNUSED void pre##_init(void) \
{ \
} \
It even almost makes sense as API: the _init function does all
initialization tasks required, which is to say, none. (The {0, ...}
initializer already has taken care of setting all fields to 0).
This can be simplified
static MAYBE_UNUSED void pre##_commit(void) \
{ \
pre##_pool.committed = pre##_pool.size; \
} \
In other words, maybe something like this on top? This includes the
vestigal _init() function which really should not be there (it is
confusing that some callers use it and others don’t). I did not
spend much time on it because in the end I suspect we might throw
obj_pool away anyway.
Oh, right. I remember that you asked to turn off persistence for this
merge. We can include persistence it in a later series.
Junio: Could you squash this diff into the commit?
$ make vcs-svn/lib.a V=1
rm -f vcs-svn/lib.a && ar rcs vcs-svn/lib.a
ar: vcs-svn/lib.a: No such file or directory
make: *** [vcs-svn/lib.a] Error 1
That is because the vcs-svn directory does not exist. So
probably the LICENSE should be added with the same patch
(and git should learn to track empty directories).
Oops. Sorry about not checking this: it looked alright at a
glance. Yes, we can add LICENSE with this patch.
Junio: Could you squash in this diff?
--- /dev/null+++ b/vcs-svn/LICENSE
@@ -0,0 +1,26 @@+Copyright (C) 2010 David Barr <david.barr@cordelta.com>.+All rights reserved.++Redistribution and use in source and binary forms, with or without+modification, are permitted provided that the following conditions+are met:+1. Redistributions of source code must retain the above copyright+ notice(s), this list of conditions and the following disclaimer+ unmodified other than the allowable addition of one or more+ copyright notices.+2. Redistributions in binary form must reproduce the above copyright+ notice(s), this list of conditions and the following disclaimer in+ the documentation and/or other materials provided with the+ distribution.++THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY+EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR+PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER(S) BE+LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR+BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,+WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE+OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,+EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.-- Ram
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:08
Ramkumar Ramachandra wrote:
From: David Barr <redacted>
This library provides thread-unsafe fgets()- and fread()-like
functions where the caller does not have to supply a buffer. It
maintains a couple of static buffers and provides an API to use
them.
NEEDSWORK: what should buffer_copy_bytes do on error?
For consistency with the rest of vcs-svn, it should do nothing. :)
I would love to see svn-fe diagnosing and recovering somehow from
faulty input. For now it follows the easier route of just ignoring
(and skipping) confusing input.
Probably this should be mentioned in the man page somewhere.
[...]
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:08
Ramkumar Ramachandra wrote:
svndump parses data that is in SVN dumpfile format produced by
`svnadmin dump` with the help of line_buffer and uses repo_tree and
fast_export to emit a git fast-import stream.
Probably worth mentioning the this requires a dumpfile v2 (i.e., it
does not understand the svndiff0 delta format yet).
Neat. This is a textbook example of where to use a perfect hash, but
comparing interned strings is simpler and fast enough (and the
bottlenecks are elsewhere).
Unknown properties are ignored. Adding stream comments to allow
recovering them is left as an exercise for the interested reader.
+static void handle_node(void)
+{
A simple reader does not cope well with this kind of function. It is
hard to know if it exhaustively deals with all cases.
But: with real-world repos (e.g. ASF) it works well enough.
+void svndump_read(char *url)
+{
Too long. I realize that writing a state machine can be hard in C;
maybe it would be easiest to package up the state in a struct and
have a separate function for the main loop body.
The patches I didn’t comment on all look good. I don’t think anything I did
comment on should prevent this reaching a wider audience.
Thanks for the pleasant read,
Jonathan
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:08
From: David Barr <redacted>
Add a memory pool library implemented using C macros. The
obj_pool_gen() macro creates a type-specific memory pool.
The memory pool library is distinguished from the existing specialized
allocators in alloc.c by using a contiguous block for all allocations.
This means that on one hand, long-lived pointers have to be written as
offsets, since the base address changes as the pool grows, but on the
other hand, the entire pool can be easily written to the file system.
This could allow the memory pool to persist between runs of an
application.
For the svn importer, such a facility is useful because each svn
revision can copy trees and files from any previous revision. The
relevant information for all revisions has to persist somehow to
support incremental runs.
Signed-off-by: David Barr <redacted>
Signed-off-by: Ramkumar Ramachandra <redacted>
Signed-off-by: Junio C Hamano <redacted>
Signed-off-by: Jonathan Nieder <redacted>
---
Stripped out pool_init. And added tests! There is not really
much an allocator can do, so it is fun to play around with.
.gitignore | 1 +
Makefile | 4 +-
t/t0080-vcs-svn.sh | 79 +++++++++++++++++++++++++++++++++++
test-obj-pool.c | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++
vcs-svn/obj_pool.h | 61 +++++++++++++++++++++++++++
5 files changed, 260 insertions(+), 1 deletions(-)
create mode 100755 t/t0080-vcs-svn.sh
create mode 100644 test-obj-pool.c
create mode 100644 vcs-svn/obj_pool.h
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:08
From: Jason Evans <redacted>
Provide macros to generate a type-specific treap implementation and
various functions to operate on it. It uses obj_pool.h to store memory
nodes in a treap. Previously committed nodes are never removed from
the pool; after any *_commit operation, it is assumed (correctly, in
the case of svn-fast-export) that someone else must care about them.
Treaps provide a memory-efficient binary search tree structure.
Insertion/deletion/search are about as about as fast in the average
case as red-black trees and the chances of worst-case behavior are
vanishingly small, thanks to (pseudo-)randomness. The bad worst-case
behavior is a small price to pay, given that treaps are much simpler
to implement.
From http://www.canonware.com/download/trp/trp_hash/trp.h
[db: Altered to reference nodes by offset from a common base pointer]
[db: Bob Jenkins' hashing implementation dropped for Knuth's]
[db: Methods unnecessary for search and insert dropped]
Signed-off-by: David Barr <redacted>
Signed-off-by: Ramkumar Ramachandra <redacted>
Signed-off-by: Junio C Hamano <redacted>
Signed-off-by: Jonathan Nieder <redacted>
Signed-off-by: Ramkumar Ramachandra <redacted>
Signed-off-by: Jonathan Nieder <redacted>
---
Tweaked treap_search() to always return the node after a missing
node, like it is documented to. For vcs-svn this doesn’t matter
but the predictable semantics should make debugging easier.
The rest of the patches are almost identical to the versions Ram
sent; see the aforementioned git tree if you are interested in
trying them out. Testing would be quite welcome.
.gitignore | 1 +
Makefile | 3 +-
t/t0080-vcs-svn.sh | 22 +++++
test-treap.c | 65 +++++++++++++++
vcs-svn/LICENSE | 3 +
vcs-svn/trp.h | 223 ++++++++++++++++++++++++++++++++++++++++++++++++++++
vcs-svn/trp.txt | 98 +++++++++++++++++++++++
7 files changed, 414 insertions(+), 1 deletions(-)
create mode 100644 test-treap.c
create mode 100644 vcs-svn/trp.h
create mode 100644 vcs-svn/trp.txt
@@ -0,0 +1,65 @@+/*+*test-treap.c:codetoexercisethesvnimporter'streapstructure+*/++#include"cache.h"+#include"vcs-svn/obj_pool.h"+#include"vcs-svn/trp.h"++structint_node{+uintmax_tn;+structtrp_nodechildren;+};++obj_pool_gen(node,structint_node,3)++staticintnode_cmp(structint_node*a,structint_node*b)+{+return(a->n>b->n)-(a->n<b->n);+}++trp_gen(static,treap_,structint_node,children,node,node_cmp)++staticvoidstrtonode(structint_node*item,constchar*s)+{+char*end;+item->n=strtoumax(s,&end,10);+if(*s=='\0'||(*end!='\n'&&*end!='\0'))+die("invalid integer: %s",s);+}++intmain(intargc,char*argv[])+{+structstrbufsb=STRBUF_INIT;+structtrp_rootroot={~0};+uint32_titem;++if(argc!=1)+usage("test-treap < ints");++while(strbuf_getline(&sb,stdin,'\n')!=EOF){+item=node_alloc(1);+strtonode(node_pointer(item),sb.buf);+treap_insert(&root,node_pointer(item));+}++item=node_offset(treap_first(&root));+while(~item){+uint32_tnext;+structint_node*tmp=node_pointer(node_alloc(1));++tmp->n=node_pointer(item)->n;+next=node_offset(treap_next(&root,node_pointer(item)));++treap_remove(&root,node_pointer(item));+item=node_offset(treap_search(&root,tmp));++if(item!=next&&(!~item||node_pointer(item)->n!=tmp->n))+die("found %"PRIuMAX" in place of %"PRIuMAX"",+~item?node_pointer(item)->n:~(uintmax_t)0,+~next?node_pointer(next)->n:~(uintmax_t)0);+printf("%"PRIuMAX"\n",tmp->n);+}+node_reset();+return0;+}
@@ -1,6 +1,9 @@ Copyright (C) 2010 David Barr <david.barr@cordelta.com>. All rights reserved.+Copyright (C) 2008 Jason Evans <jasone@canonware.com>.+All rights reserved.+ Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
@@ -0,0 +1,98 @@+Motivation+==========++Treaps provide a memory-efficient binary search tree structure.+Insertion/deletion/search are about as about as fast in the average+case as red-black trees and the chances of worst-case behavior are+vanishingly small, thanks to (pseudo-)randomness. The bad worst-case+behavior is a small price to pay, given that treaps are much simpler+to implement.++API+===++The trp API generates a data structure and functions to handle a+large growing set of objects stored in a pool.++The caller:++. Specifies parameters for the generated functions with the+ trp_gen(static, foo_, ...) macro.++. Allocates a `struct trp_root` variable and sets it to {~0}.++. Adds new nodes to the set using `foo_insert`.++. Can find a specific item in the set using `foo_search`.++. Can iterate over items in the set using `foo_first` and `foo_next`.++. Can remove an item from the set using `foo_remove`.++Example:++----+struct ex_node {+ const char *s;+ struct trp_node ex_link;+};+static struct trp_root ex_base = {~0};+obj_pool_gen(ex, struct ex_node, 4096);+trp_gen(static, ex_, struct ex_node, ex_link, ex, strcmp)+struct ex_node *item;++item = ex_pointer(ex_alloc(1));+item->s = "hello";+ex_insert(&ex_base, item);+item = ex_pointer(ex_alloc(1));+item->s = "goodbye";+ex_insert(&ex_base, item);+for (item = ex_first(&ex_base); item; item = ex_next(&ex_base, item))+ printf("%s\n", item->s);+----++Functions+---------++trp_gen(attr, foo_, node_type, link_field, pool, cmp)::++ Generate a type-specific treap implementation.+++. The storage class for generated functions will be 'attr' (e.g., `static`).+. Generated function names are prefixed with 'foo_' (e.g., `treap_`).+. Treap nodes will be of type 'node_type' (e.g., `struct treap_node`).+ This type must be a struct with at least one `struct trp_node` field+ to point to its children.+. The field used to access child nodes will be 'link_field'.+. All treap nodes must lie in the 'pool' object pool.+. Treap nodes must be totally ordered by the 'cmp' relation, with the+ following prototype:+++int (*cmp)(node_type \*a, node_type \*b)+++and returning a value less than, equal to, or greater than zero+according to the result of comparison.++void foo_insert(struct trp_root *treap, node_type \*node)::++ Insert node into treap. If inserted multiple times,+ a node will appear in the treap multiple times.++void foo_remove(struct trp_root *treap, node_type \*node)::++ Remove node from treap. Caller must ensure node is+ present in treap before using this function.++node_type *foo_search(struct trp_root \*treap, node_type \*key)::++ Search for a node that matches key. If no match is found,+ return what would be key's successor, were key in treap+ (NULL if no successor).++node_type *foo_first(struct trp_root \*treap)::++ Find the first item from the treap, in sorted order.++node_type *foo_next(struct trp_root \*treap, node_type \*node)::++ Find the next item.
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:08
Jonathan Nieder wrote:
Tweaked treap_search() to always return the node after a missing
node, like it is documented to.
In this case, the documentation was wrong.
For vcs-svn this doesn’t matter
Or rather, it does. Sorry about that.
-- 8< --
Subject: vcs-svn: treap_search should return NULL for missing items
In a misguided attempt to make the code match the documentation,
commit 4692f8e7d (Add treap implementation, 2010-07-15) changed
the semantics of treap_search to return the /next/ node when a
node is missing.
That is great in some circumstances (and the new tests even rely on
it), but the rest of vcs-svn relies on treap_search to return
NULL in that case instead. The documentation only suggested
otherwise because of a typo.
So fix it: now treap_search can do what it was always supposed
to (return NULL on failure) and Jason Evans’s treap_nsearch function
can be used to keep the test suite working.
Signed-off-by: Jonathan Nieder <redacted>
---
test-treap.c | 2 +-
vcs-svn/trp.h | 13 +++++++++++++
vcs-svn/trp.txt | 9 +++++++--
3 files changed, 21 insertions(+), 3 deletions(-)
@@ -52,7 +52,7 @@ int main(int argc, char *argv[])next=node_offset(treap_next(&root,node_pointer(item)));treap_remove(&root,node_pointer(item));-item=node_offset(treap_search(&root,tmp));+item=node_offset(treap_nsearch(&root,tmp));if(item!=next&&(!~item||node_pointer(item)->n!=tmp->n))die("found %"PRIuMAX" in place of %"PRIuMAX"",
@@ -86,8 +86,13 @@ void foo_remove(struct trp_root *treap, node_type \*node):: node_type *foo_search(struct trp_root \*treap, node_type \*key):: Search for a node that matches key. If no match is found,- return what would be key's successor, were key in treap- (NULL if no successor).+ result is NULL.++node_type *foo_nsearch(struct trp_root \*treap, node_type \*key)::++ Like `foo_search`, but if if the key is missing return what+ would be key's successor, were key in treap (NULL if no+ successor). node_type *foo_first(struct trp_root \*treap)::
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:16
approxidate() is not appropriate for reading machine-written dates
because it guesses instead of erroring out on malformed dates.
parse_date() is less convenient since it returns its output as a
string. So export the underlying function that writes a timestamp.
While at it, change the return value to match the usual convention:
return 0 for success and -1 for failure.
Signed-off-by: Jonathan Nieder <redacted>
Acked-by: Ramkumar Ramachandra <redacted>
---
As before, I think this improves code clarity, independently of
its use for svn-fe. So I would not be unhappy if it is applied
as a separate topic.
No change from last round.
cache.h | 1 +
date.c | 14 ++++++--------
2 files changed, 7 insertions(+), 8 deletions(-)
@@ -811,6 +811,7 @@ const char *show_date_relative(unsigned long time, int tz,char*timebuf,size_ttimebuf_size);intparse_date(constchar*date,char*buf,intbufsize);+intparse_date_basic(constchar*date,unsignedlong*timestamp,int*offset);voiddatestamp(char*buf,intbufsize);#define approxidate(s) approxidate_careful((s), NULL)unsignedlongapproxidate_careful(constchar*,int*);
@@ -586,7 +586,7 @@ static int date_string(unsigned long date, int offset, char *buf, int len)/* Gr. strptime is crap for this; it doesn't have a way to require RFC2822(i.e.English)day/monthnames,anditdoesn'tworkcorrectlywith%z.*/-intparse_date_toffset(constchar*date,unsignedlong*timestamp,int*offset)+intparse_date_basic(constchar*date,unsignedlong*timestamp,int*offset){structtmtm;inttm_gmt;
@@ -642,17 +642,16 @@ int parse_date_toffset(const char *date, unsigned long *timestamp, int *offset)if(!tm_gmt)*timestamp-=*offset*60;-return1;/* success */+return0;/* success */}intparse_date(constchar*date,char*result,intmaxlen){unsignedlongtimestamp;intoffset;-if(parse_date_toffset(date,×tamp,&offset)>0)-returndate_string(timestamp,offset,result,maxlen);-else+if(parse_date_basic(date,×tamp,&offset))return-1;+returndate_string(timestamp,offset,result,maxlen);}enumdate_modeparse_date_format(constchar*format)
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:16
Teach the build system to build a separate library for the
upcoming subversion interop support.
The resulting vcs-svn/lib.a does not contain any code, nor is
it built during a normal build. This is just scaffolding for
later changes.
Signed-off-by: Jonathan Nieder <redacted>
Signed-off-by: Ramkumar Ramachandra <redacted>
Signed-off-by: Junio C Hamano <redacted>
Signed-off-by: Jonathan Nieder <redacted>
---
This is just for reference; no change from last round.
Makefile | 8 +++++++-
vcs-svn/LICENSE | 26 ++++++++++++++++++++++++++
2 files changed, 33 insertions(+), 1 deletions(-)
create mode 100644 vcs-svn/LICENSE
@@ -0,0 +1,26 @@+Copyright (C) 2010 David Barr <david.barr@cordelta.com>.+All rights reserved.++Redistribution and use in source and binary forms, with or without+modification, are permitted provided that the following conditions+are met:+1. Redistributions of source code must retain the above copyright+ notice(s), this list of conditions and the following disclaimer+ unmodified other than the allowable addition of one or more+ copyright notices.+2. Redistributions in binary form must reproduce the above copyright+ notice(s), this list of conditions and the following disclaimer in+ the documentation and/or other materials provided with the+ distribution.++THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY+EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR+PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER(S) BE+LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR+BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,+WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE+OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,+EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:16
From: David Barr <redacted>
Add a memory pool library implemented using C macros. The
obj_pool_gen() macro creates a type-specific memory pool.
The memory pool library is distinguished from the existing specialized
allocators in alloc.c by using a contiguous block for all allocations.
This means that on one hand, long-lived pointers have to be written as
offsets, since the base address changes as the pool grows, but on the
other hand, the entire pool can be easily written to the file system.
This could allow the memory pool to persist between runs of an
application.
For the svn importer, such a facility is useful because each svn
revision can copy trees and files from any previous revision. The
relevant information for all revisions has to persist somehow to
support incremental runs.
[rr: minor cleanups]
[jn: added tests; removed file system backing for now]
Signed-off-by: David Barr <redacted>
Signed-off-by: Ramkumar Ramachandra <redacted>
Signed-off-by: Jonathan Nieder <redacted>
Signed-off-by: Junio C Hamano <redacted>
---
The only change from last round is the notes at the end of the commit
message. Hopefully David is less likely to be blamed for bugs I
introduced this way. :)
.gitignore | 1 +
Makefile | 4 +-
t/t0080-vcs-svn.sh | 79 +++++++++++++++++++++++++++++++++++
test-obj-pool.c | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++
vcs-svn/obj_pool.h | 61 +++++++++++++++++++++++++++
5 files changed, 260 insertions(+), 1 deletions(-)
create mode 100755 t/t0080-vcs-svn.sh
create mode 100644 test-obj-pool.c
create mode 100644 vcs-svn/obj_pool.h
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:16
From: Jason Evans <redacted>
Provide macros to generate a type-specific treap implementation and
various functions to operate on it. It uses obj_pool.h to store memory
nodes in a treap. Previously committed nodes are never removed from
the pool; after any *_commit operation, it is assumed (correctly, in
the case of svn-fast-export) that someone else must care about them.
Treaps provide a memory-efficient binary search tree structure.
Insertion/deletion/search are about as about as fast in the average
case as red-black trees and the chances of worst-case behavior are
vanishingly small, thanks to (pseudo-)randomness. The bad worst-case
behavior is a small price to pay, given that treaps are much simpler
to implement.
From http://www.canonware.com/download/trp/trp_hash/trp.h
[db: Altered to reference nodes by offset from a common base pointer]
[db: Bob Jenkins' hashing implementation dropped for Knuth's]
[db: Methods unnecessary for search and insert dropped]
[rr: Squelched compiler warnings]
[db: Added support for immutable treap nodes]
[jn: Reintroduced treap_nsearch(); with tests]
Signed-off-by: David Barr <redacted>
Signed-off-by: Ramkumar Ramachandra <redacted>
Signed-off-by: Jonathan Nieder <redacted>
---
With the treap_nsearch() fixup from last time squashed in and some
more history in the log message.
.gitignore | 1 +
Makefile | 3 +-
t/t0080-vcs-svn.sh | 22 +++++
test-treap.c | 65 ++++++++++++++
vcs-svn/LICENSE | 3 +
vcs-svn/trp.h | 236 ++++++++++++++++++++++++++++++++++++++++++++++++++++
vcs-svn/trp.txt | 103 +++++++++++++++++++++++
7 files changed, 432 insertions(+), 1 deletions(-)
create mode 100644 test-treap.c
create mode 100644 vcs-svn/trp.h
create mode 100644 vcs-svn/trp.txt
@@ -0,0 +1,65 @@+/*+*test-treap.c:codetoexercisethesvnimporter'streapstructure+*/++#include"cache.h"+#include"vcs-svn/obj_pool.h"+#include"vcs-svn/trp.h"++structint_node{+uintmax_tn;+structtrp_nodechildren;+};++obj_pool_gen(node,structint_node,3)++staticintnode_cmp(structint_node*a,structint_node*b)+{+return(a->n>b->n)-(a->n<b->n);+}++trp_gen(static,treap_,structint_node,children,node,node_cmp)++staticvoidstrtonode(structint_node*item,constchar*s)+{+char*end;+item->n=strtoumax(s,&end,10);+if(*s=='\0'||(*end!='\n'&&*end!='\0'))+die("invalid integer: %s",s);+}++intmain(intargc,char*argv[])+{+structstrbufsb=STRBUF_INIT;+structtrp_rootroot={~0};+uint32_titem;++if(argc!=1)+usage("test-treap < ints");++while(strbuf_getline(&sb,stdin,'\n')!=EOF){+item=node_alloc(1);+strtonode(node_pointer(item),sb.buf);+treap_insert(&root,node_pointer(item));+}++item=node_offset(treap_first(&root));+while(~item){+uint32_tnext;+structint_node*tmp=node_pointer(node_alloc(1));++tmp->n=node_pointer(item)->n;+next=node_offset(treap_next(&root,node_pointer(item)));++treap_remove(&root,node_pointer(item));+item=node_offset(treap_nsearch(&root,tmp));++if(item!=next&&(!~item||node_pointer(item)->n!=tmp->n))+die("found %"PRIuMAX" in place of %"PRIuMAX"",+~item?node_pointer(item)->n:~(uintmax_t)0,+~next?node_pointer(next)->n:~(uintmax_t)0);+printf("%"PRIuMAX"\n",tmp->n);+}+node_reset();+return0;+}
@@ -1,6 +1,9 @@ Copyright (C) 2010 David Barr <david.barr@cordelta.com>. All rights reserved.+Copyright (C) 2008 Jason Evans <jasone@canonware.com>.+All rights reserved.+ Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
@@ -0,0 +1,103 @@+Motivation+==========++Treaps provide a memory-efficient binary search tree structure.+Insertion/deletion/search are about as about as fast in the average+case as red-black trees and the chances of worst-case behavior are+vanishingly small, thanks to (pseudo-)randomness. The bad worst-case+behavior is a small price to pay, given that treaps are much simpler+to implement.++API+===++The trp API generates a data structure and functions to handle a+large growing set of objects stored in a pool.++The caller:++. Specifies parameters for the generated functions with the+ trp_gen(static, foo_, ...) macro.++. Allocates a `struct trp_root` variable and sets it to {~0}.++. Adds new nodes to the set using `foo_insert`.++. Can find a specific item in the set using `foo_search`.++. Can iterate over items in the set using `foo_first` and `foo_next`.++. Can remove an item from the set using `foo_remove`.++Example:++----+struct ex_node {+ const char *s;+ struct trp_node ex_link;+};+static struct trp_root ex_base = {~0};+obj_pool_gen(ex, struct ex_node, 4096);+trp_gen(static, ex_, struct ex_node, ex_link, ex, strcmp)+struct ex_node *item;++item = ex_pointer(ex_alloc(1));+item->s = "hello";+ex_insert(&ex_base, item);+item = ex_pointer(ex_alloc(1));+item->s = "goodbye";+ex_insert(&ex_base, item);+for (item = ex_first(&ex_base); item; item = ex_next(&ex_base, item))+ printf("%s\n", item->s);+----++Functions+---------++trp_gen(attr, foo_, node_type, link_field, pool, cmp)::++ Generate a type-specific treap implementation.+++. The storage class for generated functions will be 'attr' (e.g., `static`).+. Generated function names are prefixed with 'foo_' (e.g., `treap_`).+. Treap nodes will be of type 'node_type' (e.g., `struct treap_node`).+ This type must be a struct with at least one `struct trp_node` field+ to point to its children.+. The field used to access child nodes will be 'link_field'.+. All treap nodes must lie in the 'pool' object pool.+. Treap nodes must be totally ordered by the 'cmp' relation, with the+ following prototype:+++int (*cmp)(node_type \*a, node_type \*b)+++and returning a value less than, equal to, or greater than zero+according to the result of comparison.++void foo_insert(struct trp_root *treap, node_type \*node)::++ Insert node into treap. If inserted multiple times,+ a node will appear in the treap multiple times.++void foo_remove(struct trp_root *treap, node_type \*node)::++ Remove node from treap. Caller must ensure node is+ present in treap before using this function.++node_type *foo_search(struct trp_root \*treap, node_type \*key)::++ Search for a node that matches key. If no match is found,+ result is NULL.++node_type *foo_nsearch(struct trp_root \*treap, node_type \*key)::++ Like `foo_search`, but if if the key is missing return what+ would be key's successor, were key in treap (NULL if no+ successor).++node_type *foo_first(struct trp_root \*treap)::++ Find the first item from the treap, in sorted order.++node_type *foo_next(struct trp_root \*treap, node_type \*node)::++ Find the next item.
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:16
From: David Barr <redacted>
Intern strings so they can be compared by address and stored without
wasting space.
This library uses the macros in the obj_pool.h and trp.h to create a
memory pool for strings and expose an API for handling them.
[rr: added API docs]
[jn: with some API simplifications, new documentation and tests]
Signed-off-by: David Barr <redacted>
Signed-off-by: Ramkumar Ramachandra <redacted>
Signed-off-by: Jonathan Nieder <redacted>
---
New test. The return value from pool_tok_seq is not checked by
the vcs-svn lib but trying to use it in tests revealed it was not
so intuitive. pool_tok_seq() was behaving strangely when passed
an array of size 0; I think there is nothing sane to do in that
case --- maybe it should abort(). The API was passing around
char * that cannot be modified; changed to const char *.
Another set of eyes on this would be welcome.
.gitignore | 1 +
Makefile | 9 +++-
t/t0080-vcs-svn.sh | 16 +++++++
test-string-pool.c | 31 ++++++++++++++
vcs-svn/string_pool.c | 102 +++++++++++++++++++++++++++++++++++++++++++++++
vcs-svn/string_pool.h | 11 +++++
vcs-svn/string_pool.txt | 43 ++++++++++++++++++++
7 files changed, 210 insertions(+), 3 deletions(-)
create mode 100644 test-string-pool.c
create mode 100644 vcs-svn/string_pool.c
create mode 100644 vcs-svn/string_pool.h
create mode 100644 vcs-svn/string_pool.txt
@@ -0,0 +1,31 @@+/*+*test-string-pool.c:codetoexercisethesvnimporter'sstringpool+*/++#include"git-compat-util.h"+#include"vcs-svn/string_pool.h"++intmain(intargc,char*argv[])+{+constuint32_tunequal=pool_intern("does not equal");+constuint32_tequal=pool_intern("equals");+uint32_tbuf[3];+uint32_tn;++if(argc!=2)+usage("test-string-pool <string>,<string>");++n=pool_tok_seq(3,buf,",-",argv[1]);+if(n>=3)+die("too many strings");+if(n<=1)+die("too few strings");++buf[2]=buf[1];+buf[1]=(buf[0]==buf[2])?equal:unequal;+pool_print_seq(3,buf,' ',stdout);+fputc('\n',stdout);++pool_reset();+return0;+}
@@ -0,0 +1,102 @@+/*+*Licensedunderatwo-clauseBSD-stylelicense.+*SeeLICENSEfordetails.+*/++#include"git-compat-util.h"+#include"trp.h"+#include"obj_pool.h"+#include"string_pool.h"++staticstructtrp_roottree={~0};++structnode{+uint32_toffset;+structtrp_nodechildren;+};++/* Two memory pools: one for struct node, and another for strings */+obj_pool_gen(node,structnode,4096)+obj_pool_gen(string,char,4096)++staticchar*node_value(structnode*node)+{+returnnode?string_pointer(node->offset):NULL;+}++staticintnode_cmp(structnode*a,structnode*b)+{+returnstrcmp(node_value(a),node_value(b));+}++/* Build a Treap from the node structure (a trp_node w/ offset) */+trp_gen(static,tree_,structnode,children,node,node_cmp);++constchar*pool_fetch(uint32_tentry)+{+returnnode_value(node_pointer(entry));+}++uint32_tpool_intern(constchar*key)+{+/* Canonicalize key */+structnode*match=NULL;+uint32_tkey_len;+if(key==NULL)+return~0;+key_len=strlen(key)+1;+structnode*node=node_pointer(node_alloc(1));+node->offset=string_alloc(key_len);+strcpy(node_value(node),key);+match=tree_search(&tree,node);+if(!match){+tree_insert(&tree,node);+}else{+node_free(1);+string_free(key_len);+node=match;+}+returnnode_offset(node);+}++uint32_tpool_tok_r(char*str,constchar*delim,char**saveptr)+{+char*token=strtok_r(str,delim,saveptr);+returntoken?pool_intern(token):~0;+}++voidpool_print_seq(uint32_tlen,uint32_t*seq,chardelim,FILE*stream)+{+uint32_ti;+for(i=0;i<len&&~seq[i];i++){+fputs(pool_fetch(seq[i]),stream);+if(i<len-1&&~seq[i+1])+fputc(delim,stream);+}+}++uint32_tpool_tok_seq(uint32_tsz,uint32_t*seq,constchar*delim,char*str)+{+char*context=NULL;+uint32_ttoken=~0;+uint32_tlength;++if(sz==0)+return~0;+if(str)+token=pool_tok_r(str,delim,&context);+for(length=0;length<sz;length++){+seq[length]=token;+if(token==~0)+returnlength;+token=pool_tok_r(NULL,delim,&context);+}+seq[sz-1]=~0;+returnsz;+}++voidpool_reset(void)+{+node_reset();+string_reset();+}
@@ -0,0 +1,43 @@+string_pool API+===============++The string_pool API provides facilities for replacing strings+with integer keys that can be more easily compared and stored.+The facilities are designed so that one could teach Git without+too much trouble to store the information needed for these keys to+remain valid over multiple executions.++Functions+---------++pool_intern::+ Include a string in the string pool and get its key.+ If that string is already in the pool, retrieves its+ existing key.++pool_fetch::+ Retrieve the string associated to a given key.++pool_tok_r::+ Extract the key of the next token from a string.+ Interface mimics strtok_r.++pool_print_seq::+ Print a sequence of strings named by key to a file, using the+ specified delimiter to separate them.++ If NULL (key ~0) appears in the sequence, the sequence ends+ early.++pool_tok_seq::+ Split a string into tokens, storing the keys of segments+ into a caller-provided array.++ Unless sz is 0, the array will always be ~0-terminated.+ If there is not enough room for all the tokens, the+ array holds as many tokens as fit in the entries before+ the terminating ~0. Return value is the index after the+ last token, or sz if the tokens did not fit.++pool_reset::+ Deallocate storage for the string pool.
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:16
From: David Barr <redacted>
This library provides thread-unsafe fgets()- and fread()-like
functions where the caller does not have to supply a buffer. It
maintains a couple of static buffers and provides an API to use
them.
[rr: allow input from files other than stdin]
[jn: with tests, documentation, and error handling improvements]
Signed-off-by: David Barr <redacted>
Signed-off-by: Ramkumar Ramachandra <redacted>
Signed-off-by: Jonathan Nieder <redacted>
---
New tests and API docs. The return value from buffer_deinit
can be used to check for errors now (I found this useful when
writing tests).
.gitignore | 1 +
Makefile | 8 +++-
t/t0080-vcs-svn.sh | 54 ++++++++++++++++++++++++++
test-line-buffer.c | 46 ++++++++++++++++++++++
vcs-svn/line_buffer.c | 97 +++++++++++++++++++++++++++++++++++++++++++++++
vcs-svn/line_buffer.h | 12 ++++++
vcs-svn/line_buffer.txt | 58 ++++++++++++++++++++++++++++
7 files changed, 274 insertions(+), 2 deletions(-)
create mode 100644 test-line-buffer.c
create mode 100644 vcs-svn/line_buffer.c
create mode 100644 vcs-svn/line_buffer.h
create mode 100644 vcs-svn/line_buffer.txt
@@ -0,0 +1,97 @@+/*+*Licensedunderatwo-clauseBSD-stylelicense.+*SeeLICENSEfordetails.+*/++#include"git-compat-util.h"+#include"line_buffer.h"+#include"obj_pool.h"++#define LINE_BUFFER_LEN 10000+#define COPY_BUFFER_LEN 4096++/* Create memory pool for char sequence of known length */+obj_pool_gen(blob,char,4096)++staticcharline_buffer[LINE_BUFFER_LEN];+staticcharbyte_buffer[COPY_BUFFER_LEN];+staticFILE*infile;++intbuffer_init(constchar*filename)+{+infile=filename?fopen(filename,"r"):stdin;+if(!infile)+return-1;+return0;+}++intbuffer_deinit(void)+{+interr;+if(infile==stdin)+returnferror(infile);+err=ferror(infile);+err|=fclose(infile);+returnerr;+}++/* Read a line without trailing newline. */+char*buffer_read_line(void)+{+char*end;+if(!fgets(line_buffer,sizeof(line_buffer),infile))+/* Error or data exhausted. */+returnNULL;+end=line_buffer+strlen(line_buffer);+if(end[-1]=='\n')+end[-1]='\0';+elseif(feof(infile))+;/* No newline at end of file. That's fine. */+else+/*+*Linewastoolong.+*Thereisprobablyasanerwaytodealwiththis,+*butfornowlet'sreturnanerror.+*/+returnNULL;+returnline_buffer;+}++char*buffer_read_string(uint32_tlen)+{+char*s;+blob_free(blob_pool.size);+s=blob_pointer(blob_alloc(len+1));+s[fread(s,1,len,infile)]='\0';+returnferror(infile)?NULL:s;+}++voidbuffer_copy_bytes(uint32_tlen)+{+uint32_tin;+while(len>0&&!feof(infile)&&!ferror(infile)){+in=len<COPY_BUFFER_LEN?len:COPY_BUFFER_LEN;+in=fread(byte_buffer,1,in,infile);+len-=in;+fwrite(byte_buffer,1,in,stdout);+if(ferror(stdout)){+buffer_skip_bytes(len);+return;+}+}+}++voidbuffer_skip_bytes(uint32_tlen)+{+uint32_tin;+while(len>0&&!feof(infile)&&!ferror(infile)){+in=len<COPY_BUFFER_LEN?len:COPY_BUFFER_LEN;+in=fread(byte_buffer,1,in,infile);+len-=in;+}+}++voidbuffer_reset(void)+{+blob_reset();+}
@@ -0,0 +1,58 @@+line_buffer API+===============++The line_buffer library provides a convenient interface for+mostly-line-oriented input.++Each line is not permitted to exceed 10000 bytes. The provided+functions are not thread-safe or async-signal-safe, and like+`fgets()`, they generally do not function correctly if interrupted+by a signal without SA_RESTART set.++Calling sequence+----------------++The calling program:++ - specifies a file to read with `buffer_init`+ - processes input with `buffer_read_line`, `buffer_read_string`,+ `buffer_skip_bytes`, and `buffer_copy_bytes`+ - closes the file with `buffer_deinit`, perhaps to start over and+ read another file.++Before exiting, the caller can use `buffer_reset` to deallocate+resources for the benefit of profiling tools.++Functions+---------++`buffer_init`::+ Open the named file for input. If filename is NULL,+ start reading from stdin. On failure, returns -1 (with+ errno indicating the nature of the failure).++`buffer_deinit`::+ Stop reading from the current file (closing it unless+ it was stdin). Returns nonzero if `fclose` fails or+ the error indicator was set.++`buffer_read_line`::+ Read a line and strip off the trailing newline.+ On failure or end of file, returns NULL.++`buffer_read_string`::+ Read `len` characters of input or up to the end of the+ file, whichever comes first. Returns NULL on error.+ Returns whatever characters were read (possibly "")+ for end of file.++`buffer_copy_bytes`::+ Read `len` bytes of input and dump them to the standard output+ stream. Returns early for error or end of file.++`buffer_skip_bytes`::+ Discards `len` bytes from the input stream (stopping early+ if necessary because of an error or eof).++`buffer_reset`::+ Deallocates non-static buffers.
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:16
From: David Barr <redacted>
repo_tree maintains the exporter's state and provides a facility to to
call fast_export, which writes objects to stdout suitable for
consumption by fast-import.
The exported functions roughly correspond to Subversion FS operations.
. repo_add, repo_modify, repo_copy, repo_replace, and repo_delete
update the current commit, based roughly on the corresponding
Subversion FS operation.
. repo_commit calls out to fast_export to write the current commit to
the fast-import stream in stdout.
. repo_diff is used by the fast_export module to write the changes
for a commit.
. repo_reset erases the exporter's state, so valgrind can be happy.
[rr: squelched compiler warnings]
[jn: removed support for maintaining state on-disk, though we may
want to add it back later]
Signed-off-by: David Barr <redacted>
Signed-off-by: Ramkumar Ramachandra <redacted>
Signed-off-by: Jonathan Nieder <redacted>
---
No tests for this one, since the next few patches exercise it and this
is not a general-purpose API. Relative to last round, the
`pool_commit` and `commit_commit` calls have been eliminated; unlike
the `dir_commit` et al calls, those were only meant for committing
state to disk, and the changing high-water mark was not being used.
Makefile | 5 +-
vcs-svn/fast_export.c | 74 +++++++++++
vcs-svn/fast_export.h | 11 ++
vcs-svn/repo_tree.c | 328 +++++++++++++++++++++++++++++++++++++++++++++++++
vcs-svn/repo_tree.h | 26 ++++
5 files changed, 442 insertions(+), 2 deletions(-)
create mode 100644 vcs-svn/fast_export.c
create mode 100644 vcs-svn/fast_export.h
create mode 100644 vcs-svn/repo_tree.c
create mode 100644 vcs-svn/repo_tree.h
@@ -0,0 +1,328 @@+/*+*Licensedunderatwo-clauseBSD-stylelicense.+*SeeLICENSEfordetails.+*/++#include"git-compat-util.h"++#include"string_pool.h"+#include"repo_tree.h"+#include"obj_pool.h"+#include"fast_export.h"++#include"trp.h"++structrepo_dirent{+uint32_tname_offset;+structtrp_nodechildren;+uint32_tmode;+uint32_tcontent_offset;+};++structrepo_dir{+structtrp_rootentries;+};++structrepo_commit{+uint32_troot_dir_offset;+};++/* Memory pools for commit, dir and dirent */+obj_pool_gen(commit,structrepo_commit,4096)+obj_pool_gen(dir,structrepo_dir,4096)+obj_pool_gen(dirent,structrepo_dirent,4096)++staticuint32_tactive_commit;+staticuint32_tmark;++staticintrepo_dirent_name_cmp(constvoid*a,constvoid*b);++/* Treap for directory entries */+trp_gen(static,dirent_,structrepo_dirent,children,dirent,repo_dirent_name_cmp);++uint32_tnext_blob_mark(void)+{+returnmark++;+}++staticstructrepo_dir*repo_commit_root_dir(structrepo_commit*commit)+{+returndir_pointer(commit->root_dir_offset);+}++staticstructrepo_dirent*repo_first_dirent(structrepo_dir*dir)+{+returndirent_first(&dir->entries);+}++staticintrepo_dirent_name_cmp(constvoid*a,constvoid*b)+{+conststructrepo_dirent*dirent1=a,*dirent2=b;+uint32_ta_offset=dirent1->name_offset;+uint32_tb_offset=dirent2->name_offset;+return(a_offset>b_offset)-(a_offset<b_offset);+}++staticintrepo_dirent_is_dir(structrepo_dirent*dirent)+{+returndirent!=NULL&&dirent->mode==REPO_MODE_DIR;+}++staticstructrepo_dir*repo_dir_from_dirent(structrepo_dirent*dirent)+{+if(!repo_dirent_is_dir(dirent))+returnNULL;+returndir_pointer(dirent->content_offset);+}++staticstructrepo_dir*repo_clone_dir(structrepo_dir*orig_dir)+{+uint32_torig_o,new_o;+orig_o=dir_offset(orig_dir);+if(orig_o>=dir_pool.committed)+returnorig_dir;+new_o=dir_alloc(1);+orig_dir=dir_pointer(orig_o);+*dir_pointer(new_o)=*orig_dir;+returndir_pointer(new_o);+}++staticstructrepo_dirent*repo_read_dirent(uint32_trevision,uint32_t*path)+{+uint32_tname=0;+structrepo_dirent*key=dirent_pointer(dirent_alloc(1));+structrepo_dir*dir=NULL;+structrepo_dirent*dirent=NULL;+dir=repo_commit_root_dir(commit_pointer(revision));+while(~(name=*path++)){+key->name_offset=name;+dirent=dirent_search(&dir->entries,key);+if(dirent==NULL||!repo_dirent_is_dir(dirent))+break;+dir=repo_dir_from_dirent(dirent);+}+dirent_free(1);+returndirent;+}++staticvoidrepo_write_dirent(uint32_t*path,uint32_tmode,+uint32_tcontent_offset,uint32_tdel)+{+uint32_tname,revision,dir_o=~0,parent_dir_o=~0;+structrepo_dir*dir;+structrepo_dirent*key;+structrepo_dirent*dirent=NULL;+revision=active_commit;+dir=repo_commit_root_dir(commit_pointer(revision));+dir=repo_clone_dir(dir);+commit_pointer(revision)->root_dir_offset=dir_offset(dir);+while(~(name=*path++)){+parent_dir_o=dir_offset(dir);++key=dirent_pointer(dirent_alloc(1));+key->name_offset=name;++dirent=dirent_search(&dir->entries,key);+if(dirent==NULL)+dirent=key;+else+dirent_free(1);++if(dirent==key){+dirent->mode=REPO_MODE_DIR;+dirent->content_offset=0;+dirent_insert(&dir->entries,dirent);+}++if(dirent_offset(dirent)<dirent_pool.committed){+dir_o=repo_dirent_is_dir(dirent)?+dirent->content_offset:~0;+dirent_remove(&dir->entries,dirent);+dirent=dirent_pointer(dirent_alloc(1));+dirent->name_offset=name;+dirent->mode=REPO_MODE_DIR;+dirent->content_offset=dir_o;+dirent_insert(&dir->entries,dirent);+}++dir=repo_dir_from_dirent(dirent);+dir=repo_clone_dir(dir);+dirent->content_offset=dir_offset(dir);+}+if(dirent==NULL)+return;+dirent->mode=mode;+dirent->content_offset=content_offset;+if(del&&~parent_dir_o)+dirent_remove(&dir_pointer(parent_dir_o)->entries,dirent);+}++uint32_trepo_copy(uint32_trevision,uint32_t*src,uint32_t*dst)+{+uint32_tmode=0,content_offset=0;+structrepo_dirent*src_dirent;+src_dirent=repo_read_dirent(revision,src);+if(src_dirent!=NULL){+mode=src_dirent->mode;+content_offset=src_dirent->content_offset;+repo_write_dirent(dst,mode,content_offset,0);+}+returnmode;+}++voidrepo_add(uint32_t*path,uint32_tmode,uint32_tblob_mark)+{+repo_write_dirent(path,mode,blob_mark,0);+}++uint32_trepo_replace(uint32_t*path,uint32_tblob_mark)+{+uint32_tmode=0;+structrepo_dirent*src_dirent;+src_dirent=repo_read_dirent(active_commit,path);+if(src_dirent!=NULL){+mode=src_dirent->mode;+repo_write_dirent(path,mode,blob_mark,0);+}+returnmode;+}++voidrepo_modify(uint32_t*path,uint32_tmode,uint32_tblob_mark)+{+structrepo_dirent*src_dirent;+src_dirent=repo_read_dirent(active_commit,path);+if(src_dirent!=NULL&&blob_mark==0)+blob_mark=src_dirent->content_offset;+repo_write_dirent(path,mode,blob_mark,0);+}++voidrepo_delete(uint32_t*path)+{+repo_write_dirent(path,0,0,1);+}++staticvoidrepo_git_add_r(uint32_tdepth,uint32_t*path,structrepo_dir*dir);++staticvoidrepo_git_add(uint32_tdepth,uint32_t*path,structrepo_dirent*dirent)+{+if(repo_dirent_is_dir(dirent))+repo_git_add_r(depth,path,repo_dir_from_dirent(dirent));+else+fast_export_modify(depth,path,+dirent->mode,dirent->content_offset);+}++staticvoidrepo_git_add_r(uint32_tdepth,uint32_t*path,structrepo_dir*dir)+{+structrepo_dirent*de=repo_first_dirent(dir);+while(de){+path[depth]=de->name_offset;+repo_git_add(depth+1,path,de);+de=dirent_next(&dir->entries,de);+}+}++staticvoidrepo_diff_r(uint32_tdepth,uint32_t*path,structrepo_dir*dir1,+structrepo_dir*dir2)+{+structrepo_dirent*de1,*de2;+de1=repo_first_dirent(dir1);+de2=repo_first_dirent(dir2);++while(de1&&de2){+if(de1->name_offset<de2->name_offset){+path[depth]=de1->name_offset;+fast_export_delete(depth+1,path);+de1=dirent_next(&dir1->entries,de1);+continue;+}+if(de1->name_offset>de2->name_offset){+path[depth]=de2->name_offset;+repo_git_add(depth+1,path,de2);+de2=dirent_next(&dir2->entries,de2);+continue;+}+path[depth]=de1->name_offset;++if(de1->mode==de2->mode&&+de1->content_offset==de2->content_offset){+;/* No change. */+}elseif(repo_dirent_is_dir(de1)&&repo_dirent_is_dir(de2)){+repo_diff_r(depth+1,path,+repo_dir_from_dirent(de1),+repo_dir_from_dirent(de2));+}elseif(!repo_dirent_is_dir(de1)&&!repo_dirent_is_dir(de2)){+repo_git_add(depth+1,path,de2);+}else{+fast_export_delete(depth+1,path);+repo_git_add(depth+1,path,de2);+}+de1=dirent_next(&dir1->entries,de1);+de2=dirent_next(&dir2->entries,de2);+}+while(de1){+path[depth]=de1->name_offset;+fast_export_delete(depth+1,path);+de1=dirent_next(&dir1->entries,de1);+}+while(de2){+path[depth]=de2->name_offset;+repo_git_add(depth+1,path,de2);+de2=dirent_next(&dir2->entries,de2);+}+}++staticuint32_tpath_stack[REPO_MAX_PATH_DEPTH];++voidrepo_diff(uint32_tr1,uint32_tr2)+{+repo_diff_r(0,+path_stack,+repo_commit_root_dir(commit_pointer(r1)),+repo_commit_root_dir(commit_pointer(r2)));+}++voidrepo_commit(uint32_trevision,uint32_tauthor,char*log,uint32_tuuid,+uint32_turl,unsignedlongtimestamp)+{+fast_export_commit(revision,author,log,uuid,url,timestamp);+dirent_commit();+dir_commit();+active_commit=commit_alloc(1);+commit_pointer(active_commit)->root_dir_offset=+commit_pointer(active_commit-1)->root_dir_offset;+}++staticvoidmark_init(void)+{+uint32_ti;+mark=0;+for(i=0;i<dirent_pool.size;i++)+if(!repo_dirent_is_dir(dirent_pointer(i))&&+dirent_pointer(i)->content_offset>mark)+mark=dirent_pointer(i)->content_offset;+mark++;+}++voidrepo_init(){+mark_init();+if(commit_pool.size==0){+/* Create empty tree for commit 0. */+commit_alloc(1);+commit_pointer(0)->root_dir_offset=dir_alloc(1);+dir_pointer(0)->entries.trp_root=~0;+dir_commit();+}+/* Preallocate next commit, ready for changes. */+active_commit=commit_alloc(1);+commit_pointer(active_commit)->root_dir_offset=+commit_pointer(active_commit-1)->root_dir_offset;+}++voidrepo_reset(void)+{+pool_reset();+commit_reset();+dir_reset();+dirent_reset();+}
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:16
From: David Barr <redacted>
svndump parses data that is in SVN dumpfile format produced by
`svnadmin dump` with the help of line_buffer and uses repo_tree and
fast_export to emit a git fast-import stream.
Based roughly on com.hydrografix.svndump 0.92 from the SvnToCCase
project at <http://svn2cc.sarovar.org/>, by Stefan Hegny and
others.
[rr: allow input from files other than stdin]
[jn: with test, more error reporting]
Signed-off-by: David Barr <redacted>
Signed-off-by: Ramkumar Ramachandra <redacted>
Signed-off-by: Jonathan Nieder <redacted>
---
New test. It is slow; work by svn gurus to speed this up would
be nice. The test program is very similar to svn-fe from contrib,
except it exercises Ram’s change to read from a file other than
stdin.
.gitignore | 1 +
Makefile | 8 +-
contrib/svn-fe/svn-fe.c | 1 +
t/t9010-svn-fe.sh | 32 +++++
test-svn-fe.c | 18 +++
vcs-svn/LICENSE | 4 +
vcs-svn/svndump.c | 302 +++++++++++++++++++++++++++++++++++++++++++++++
vcs-svn/svndump.h | 9 ++
8 files changed, 373 insertions(+), 2 deletions(-)
create mode 100644 t/t9010-svn-fe.sh
create mode 100644 test-svn-fe.c
create mode 100644 vcs-svn/svndump.c
create mode 100644 vcs-svn/svndump.h
@@ -4,6 +4,10 @@ All rights reserved. Copyright (C) 2008 Jason Evans <jasone@canonware.com>. All rights reserved.+Copyright (C) 2005 Stefan Hegny, hydrografix Consulting GmbH,+Frankfurt/Main, Germany+and others, see http://svn2cc.sarovar.org+ Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:16
The svn-fe example does not litter the working directory with
.bin files any more (hoorah!).
The permissive error handling implies a known bug. We should
be flagging iffy input and, even if we continue, reporting it
on exit.
Cc: David Barr <redacted>
Signed-off-by: Jonathan Nieder <redacted>
---
contrib/svn-fe/svn-fe.txt | 14 ++++++--------
1 files changed, 6 insertions(+), 8 deletions(-)
@@ -43,11 +43,9 @@ user <user@UUID> as committer, where 'user' is the value of the `svn:author` property and 'UUID' the repository's identifier.-To support incremental imports, 'svn-fe' will put a `git-svn-id`-line at the end of each commit log message if passed an url on the-command line. This line has the form `git-svn-id: URL@REVNO UUID`.--Empty directories and unknown properties are silently discarded.+To support incremental imports, 'svn-fe' puts a `git-svn-id` line at+the end of each commit log message if passed an url on the command+line. This line has the form `git-svn-id: URL@REVNO UUID`. The resulting repository will generally require further processing to put each project in its own repository and to separate the history
@@ -56,9 +54,9 @@ may be useful for this purpose. BUGS -----Litters the current working directory with .bin files for-persistence. Will be fixed when the svn-fe infrastructure is aware of-a Git working directory.+Empty directories and unknown properties are silently discarded.++The exit status does not reflect whether an error was detected. SEE ALSO --------
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:16
Those in the know would notice that dump file format version 2
means "svnadmin dump --no-deltas", but for the rest of us, an
explicit reminder is useful.
Signed-off-by: Jonathan Nieder <redacted>
---
That’s the end of the series. Thanks for reading.
On the horizon, as you may have guessed, are changes to use
dumpfile format v3. Doing so sanely requires two-way communication
with fast-import, I think (as discussed). Ram has already put
together a prototype delta applier, so it seems to be mostly a matter
of plumbing now.
contrib/svn-fe/svn-fe.txt | 5 ++++-
1 files changed, 4 insertions(+), 1 deletions(-)
@@ -12,7 +12,7 @@ svnadmin dump --incremental REPO | svn-fe [url] | git fast-import DESCRIPTION ------------Converts a Subversion dumpfile (version: 2) into input suitable for+Converts a Subversion dumpfile into input suitable for git-fast-import(1) and similar importers. REPO is a path to a Subversion repository mirrored on the local disk. Remote Subversion repositories can be mirrored on local disk using the `svnsync`
@@ -25,6 +25,9 @@ Subversion's repository dump format is documented in full in Files in this format can be generated using the 'svnadmin dump' or 'svk admin dump' command.+Dumps produced with 'svnadmin dump --deltas' (dumpfile format v3)+are not supported.+ OUTPUT FORMAT ------------- The fast-import format is documented by the git-fast-import(1)
svn-fe has some serious changes on the horizon. As a preparation,
let’s round up what we have now.
The most controversial change is probably the new svn-fe test, which
takes about 15 seconds (for the “svnadmin load”, not the svn-fe
step :)). It is in the t9* series, so hopefully that will not
dissuade people from running the earlier tests.
I'll comment on this separately.
The main highlight in the changes is a new
Input error
to stderr if a system call failed in reading in the dump file.
It still returns status 0 in this and other error situations,
though.
I'll comment on this separately.
Based on maint (for no good reason; that’s just where I tried it).
Intended to replace rr/svn-export in pu (only if Ram likes it, of
course).
Thanks for re-rolling (again)! You've also added a note to the commit
messages briefly explaining what each contributor has done. I'd
expected some incremental patches instead of a full re-roll, but
whatever works is good :)
David Barr (5):
Add memory pool library
Add string-specific memory pool
Add stream helper library
Infrastructure to write revisions in fast-export format
SVN dump parser
Jason Evans (1):
Add treap implementation
Jonathan Nieder (4):
Introduce vcs-svn lib
All these are good :)
Export parse_date_basic() to convert a date string to timestamp
Wasn't this ejected from this series and made a separate patch?
Update svn-fe manual
Removed the BUG since we've turned off persistence.
svn-fe manual: Clarify warning about deltas in dumpfiles
We have to fix this real soon- I'm waiting for the weekend so I get
some solid chunks of hacking time.
-- Ram
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:18
Junio C Hamano wrote:
Need SP after "while" (there are other occurrences).
Good catch. checkpatch also notices some long lines, but I think
that’s worth ignoring.
-- 8< --
Subject: treap: style fix
Missing spaces in while (0) and trpn_pointer(a, b).
Remove parentheses around return value.
Signed-off-by: Jonathan Nieder <redacted>
---
vcs-svn/trp.h | 30 +++++++++++++++---------------
1 files changed, 15 insertions(+), 15 deletions(-)
@@ -7,9 +7,9 @@#define say1(a,b) fprintf(stderr, a, b)#define say2(a,b,c) fprintf(stderr, a, b, c)#else-#define say(a) do {} while(0)-#define say1(a,b) do {} while(0)-#define say2(a,b,c) do {} while(0)+#define say(a) do { /* nothing */ } while (0)+#define say1(a,b) do { /* nothing */ } while (0)+#define say2(a,b,c) do { /* nothing */ } while (0)#endifstaticconstcharen85[]={
@@ -449,7 +449,7 @@ extern int init_db(const char *template_dir, unsigned int flags);alloc=alloc_nr(alloc);\x=xrealloc((x),alloc*sizeof(*(x)));\}\-}while(0)+}while(0)/* Initialize and use the cache information */externintread_index(structindex_state*);
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:45
Hi,
The svndiff format has proved more difficult to parse than expected.
This series documents the current state of things, and though it is
not complete, it should be ready for nitpicking by the masses.
Patches 1-4 modify the line_buffer API by introducing a struct
line_buffer to collect state that was previously held in global
variables. Callers can use multiple line_buffers to manage input from
multiple files at a time.
Patches 5-10 add various utility functions to the line_buffer API
(wrapping strbuf_fread(), fgetc(), etc). Putting the helpers there
instead of having callers work with the FILE* directly means one
could easily
- tweak the input stream (to insert "link: " at the beginning
for symlinks?);
- trace reads, for debugging; or
- use read() directly in place of stdio and limit the number of bytes
buffered
if one wants to.
Patch 11 adds a data structure and function to manage a "sliding
window" without using mmap() or fseek(). See the svndiff0 spec[1] for
how this would be used.
Patches 12 and 13 are some basic components for reading an svndiff0
file: reading variable-length integers and the opening magic bytes.
Patch 15 makes the svn-fe test usable on systems (like Ram's) without
libsvn-perl installed. It also should make the test easier to read
for people unfamiliar with lib-git-svn.sh.
Patch 16 is the delta parser/applier. This patch does _not_ add it to
contrib/svn-fe, even though that would be useful, since the
command-line interface is not set in stone yet. If you want to try it
out, use the test-svn-fe command:
test-svn-fe -d <preimage> <delta> <delta length>
The preimage or delta arg can be /dev/stdin for use in a pipeline.
Both are only read sequentially; they do not need to be regular files.
One of the test cases is enormous. The svn delta lib doesn't use
multiple windows except when dealing with relatively big files, but
probably the test case should be replaced with a smaller, artificial
example.
One of the test cases does not pass. I also don't know how to apply
the delta by hand --- it seems to have some extra bytes at the end. :(
Unfortunately the svndiff0 spec is not as clear about when to stop
reading as one might like
The code separately maintains nominal and actual lengths for a few
buffers, since truncated input is permitted (and even required) in the
deltas svn produces, though the svndiff0 spec does not document the
semantics of that.
For svn-fe changes to take advantage of this code to handle the
dumpfilev3 format, see <git://github.com/barrbrain/git.git>[2]. So
now the full svnrdump | svn-fe | fast-import pipeline can be
experienced. It still chokes on some deltas in the wild.
Thoughts, cleanups, test cases, bug reports, improvements welcome. :)
Enjoy,
Jonathan Nieder (15):
vcs-svn: Eliminate global byte_buffer[] array
vcs-svn: Replace buffer_read_string()'s memory pool with a strbuf
vcs-svn: Collect line_buffer data in a struct
vcs-svn: Teach line_buffer to handle multiple input files
vcs-svn: Make buffer_skip_bytes() report partial reads
vcs-svn: Better support for reading large files
vcs-svn: Add binary-safe read() function
vcs-svn: Let callers peek ahead to find stream end
vcs-svn: Allow input errors to be detected early
vcs-svn: Allow character-oriented input
vcs-svn: Add code to maintain a sliding view of a file
vcs-svn: Learn to parse variable-length integers
vcs-svn: Learn to check for SVN\0 magic
compat: helper for detecting unsigned overflow
vcs-svn: Add svn delta parser
Ramkumar Ramachandra (1):
t9010 (svn-fe): Eliminate dependency on svn perl bindings
Makefile | 5 +-
vcs-svn/line_buffer.txt | 8 +-
vcs-svn/fast_export.c | 6 +-
vcs-svn/fast_export.h | 5 +-
vcs-svn/line_buffer.c | 99 +-
vcs-svn/line_buffer.h | 29 +-
vcs-svn/sliding_window.c | 65 +
vcs-svn/sliding_window.h | 14 +
vcs-svn/svndiff.c | 344 +
vcs-svn/svndiff.h | 9 +
vcs-svn/svndump.c | 29 +-
vcs-svn/LICENSE | 2 +
git-compat-util.h | 6 +
test-line-buffer.c | 17 +-
test-svn-fe.c | 37 +-
t/t9010-svn-fe.sh | 29 +-
t/t9010/Xerces.cpp.diff0 | Bin 0 -> 12185 bytes
t/t9010/Xerces.cpp.done |54963 +++++++++++++++++++++++++++++++++++++++++++++
t/t9010/Xerces.cpp.src |55052 ++++++++++++++++++++++++++++++++++++++++++++++
t/t9010/newdata.diff0 | Bin 0 -> 19392 bytes
t/t9010/newdata.done | 522 +
t/t9010/src.diff0 | Bin 0 -> 74 bytes
t/t9010/src.done | 522 +
23 files changed, 111677 insertions(+), 86 deletions(-)
create mode 100644 vcs-svn/sliding_window.c
create mode 100644 vcs-svn/sliding_window.h
create mode 100644 vcs-svn/svndiff.c
create mode 100644 vcs-svn/svndiff.h
create mode 100644 t/t9010/Xerces.cpp.diff0
create mode 100644 t/t9010/Xerces.cpp.done
create mode 100644 t/t9010/Xerces.cpp.src
create mode 100644 t/t9010/blank.done
create mode 100644 t/t9010/newdata.diff0
create mode 100644 t/t9010/newdata.done
create mode 100644 t/t9010/src.diff0
create mode 100644 t/t9010/src.done
[1] http://svn.apache.org/repos/asf/subversion/trunk/notes/svndiff
[2] And some design notes:
http://thread.gmane.org/gmane.comp.version-control.git/150005/focus=157119
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:45
The data stored in byte_buffer[] is always either discarded or
written to stdout immediately. No need for it to persist between
function calls.
Signed-off-by: Jonathan Nieder <redacted>
---
vcs-svn/line_buffer.c | 3 ++-
1 files changed, 2 insertions(+), 1 deletions(-)
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:45
Prepare for the line_buffer lib to support input from multiple files,
by collecting global state in a struct that can be easily passed around.
Signed-off-by: Jonathan Nieder <redacted>
---
vcs-svn/line_buffer.c | 45 ++++++++++++++++++++++-----------------------
vcs-svn/line_buffer.h | 11 +++++++++++
2 files changed, 33 insertions(+), 23 deletions(-)
@@ -25,10 +24,10 @@ int buffer_init(const char *filename)intbuffer_deinit(void){interr;-if(infile==stdin)-returnferror(infile);-err=ferror(infile);-err|=fclose(infile);+if(buf->infile==stdin)+returnferror(buf->infile);+err=ferror(buf->infile);+err|=fclose(buf->infile);returnerr;}
@@ -36,13 +35,13 @@ int buffer_deinit(void)char*buffer_read_line(void){char*end;-if(!fgets(line_buffer,sizeof(line_buffer),infile))+if(!fgets(buf->line_buffer,sizeof(buf->line_buffer),buf->infile))/* Error or data exhausted. */returnNULL;-end=line_buffer+strlen(line_buffer);+end=buf->line_buffer+strlen(buf->line_buffer);if(end[-1]=='\n')end[-1]='\0';-elseif(feof(infile))+elseif(feof(buf->infile));/* No newline at end of file. That's fine. */else/*
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:45
Collect the line_buffer state in a newly public line_buffer struct.
Callers can use multiple line_buffers to manage input from multiple
files at a time.
The Subversion-format delta applier will use this to stream a delta
and the preimage it applies to at the same time.
Signed-off-by: Jonathan Nieder <redacted>
---
vcs-svn/line_buffer.txt | 5 +++--
vcs-svn/fast_export.c | 6 +++---
vcs-svn/fast_export.h | 5 ++++-
vcs-svn/line_buffer.c | 20 ++++++++------------
vcs-svn/line_buffer.h | 14 +++++++-------
vcs-svn/svndump.c | 29 ++++++++++++++++-------------
test-line-buffer.c | 17 +++++++++--------
7 files changed, 50 insertions(+), 46 deletions(-)
@@ -14,14 +14,15 @@ Calling sequence The calling program:+ - initializes a `struct line_buffer` to LINE_BUFFER_INIT - specifies a file to read with `buffer_init` - processes input with `buffer_read_line`, `buffer_read_string`, `buffer_skip_bytes`, and `buffer_copy_bytes` - closes the file with `buffer_deinit`, perhaps to start over and read another file.-Before exiting, the caller can use `buffer_reset` to deallocate-resources for the benefit of profiling tools.+When finished, the caller can use `buffer_reset` to deallocate+resources. Functions ---------
@@ -32,7 +28,7 @@ int buffer_deinit(void)}/* Read a line without trailing newline. */-char*buffer_read_line(void)+char*buffer_read_line(structline_buffer*buf){char*end;if(!fgets(buf->line_buffer,sizeof(buf->line_buffer),buf->infile))
@@ -53,7 +53,8 @@ Functions `buffer_skip_bytes`:: Discards `len` bytes from the input stream (stopping early- if necessary because of an error or eof).+ if necessary because of an error or eof). Return value is+ the number of bytes successfully read. `buffer_reset`:: Deallocates non-static buffers.
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:45
Tweak the line_buffer API to permit seeking and cat-ing segments
longer than 4 GiB. This would be particularly useful for applying
deltas that remove a large segment from the middle of a file.
Callers would still have to be updated to take advantage of this.
Signed-off-by: Jonathan Nieder <redacted>
---
Since off_t is a signed type, on systems with 32-bit file offsets,
this might make things worse. Is that worth worrying about?
vcs-svn/line_buffer.c | 8 ++++----
vcs-svn/line_buffer.h | 4 ++--
2 files changed, 6 insertions(+), 6 deletions(-)
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:45
buffer_read_binary() writes to a strbuf so the caller does not need
to keep track of the number of bytes read.
Signed-off-by: Jonathan Nieder <redacted>
---
vcs-svn/line_buffer.c | 6 ++++++
vcs-svn/line_buffer.h | 1 +
2 files changed, 7 insertions(+), 0 deletions(-)
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:45
The buffer_at_eof() function returns 1 if and only if all input from
the input stream has been exhausted (because of EOF or error). The
implementation calls fgetc() followed by ungetc() to force an EOF
condition when there is no more input remaining.
Like many functions in the line_buffer API, this function is not
thread-safe. It could be made to be so with a mutex if needed.
Signed-off-by: Jonathan Nieder <redacted>
---
vcs-svn/line_buffer.c | 10 ++++++++++
vcs-svn/line_buffer.h | 1 +
2 files changed, 11 insertions(+), 0 deletions(-)
@@ -27,6 +27,16 @@ int buffer_deinit(struct line_buffer *buf)returnerr;}+intbuffer_at_eof(structline_buffer*buf)+{+intch;+if((ch=fgetc(buf->infile))==EOF)+return1;+if(ungetc(ch,buf->infile)==EOF)+returnerror("cannot unget %c: %s\n",ch,strerror(errno));+return0;+}+/* Read a line without trailing newline. */char*buffer_read_line(structline_buffer*buf){
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:45
Add a buffer_ferror() function to read the error flag from the input
stream, so callers can do:
some_error_prone_operation(f, ...);
if (buffer_ferror(f))
return error("input error: %s", strerror(errno));
instead of waiting until it is time to close the file.
Signed-off-by: Jonathan Nieder <redacted>
---
vcs-svn/line_buffer.c | 5 +++++
vcs-svn/line_buffer.h | 1 +
2 files changed, 6 insertions(+), 0 deletions(-)
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:45
buffer_read_char() can be used in place of buffer_read_string(1)
to avoid consuming valuable static buffer space. The delta applier
will use this to read variable-length integers one byte at a time.
Underneath, it is fgetc(), wrapped so the line_buffer library can
maintain its role as gatekeeper of input.
Signed-off-by: Jonathan Nieder <redacted>
---
vcs-svn/line_buffer.c | 5 +++++
vcs-svn/line_buffer.h | 1 +
2 files changed, 6 insertions(+), 0 deletions(-)
@@ -42,6 +42,11 @@ int buffer_at_eof(struct line_buffer *buf)return0;}+intbuffer_read_char(structline_buffer*buf)+{+returnfgetc(buf->infile);+}+/* Read a line without trailing newline. */char*buffer_read_line(structline_buffer*buf){
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:45
Subversion's delta format has the convenient property that applying
each section of the delta only requires examining (and keeping in
memory) a small portion of the preimage. At any moment, this portion
begins at a well-defined file offset and has a well-defined length,
and as the delta is applied, it moves from the beginning to the end
of the file. Add a move_window() function to keep track of such a
window into a file.
You can use it like this:
struct line_buffer preimage = LINE_BUFFER_INIT;
buffer_init(&preimage, NULL);
struct view window = {&preimage, 0, STRBUF_INIT};
move_window(&window, 3, 7); /* (1) */
move_window(&window, 5, 5); /* (2) */
move_window(&window, 12, 2); /* (3) */
strbuf_release(&window.buf);
buffer_deinit(&preimage);
In this example: (1) reads 10 bytes and discards the first 3;
(2) discards the first 2, which are not needed any more; and (3)
skips 2 bytes and reads 2 new bytes to work with.
Whenever move_window() returns, the file position indicator is at
position window->off + window->buf.len and the data from positions
window->off to the current file position are stored in window->buf.
This function does only sequential access and never seeks, so it
can be safely used on pipes and sockets.
On end-of-file, move_window() just silently reads less than the
caller requested. On other errors, it prints a message to stderr
and returns -1.
Helped-by: David Barr [off-list ref]
Signed-off-by: Jonathan Nieder <redacted>
---
Makefile | 5 ++-
vcs-svn/sliding_window.c | 65 ++++++++++++++++++++++++++++++++++++++++++++++
vcs-svn/sliding_window.h | 14 ++++++++++
vcs-svn/LICENSE | 2 +
4 files changed, 84 insertions(+), 2 deletions(-)
create mode 100644 vcs-svn/sliding_window.c
create mode 100644 vcs-svn/sliding_window.h
@@ -1,6 +1,8 @@ Copyright (C) 2010 David Barr <david.barr@cordelta.com>. All rights reserved.+Copyright (C) 2010 Jonathan Nieder <jrnieder@gmail.com>.+ Copyright (C) 2008 Jason Evans <jasone@canonware.com>. All rights reserved.
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:45
The humble beginnings of the svn-format delta applier.
Signed-off-by: Jonathan Nieder <redacted>
---
Maybe this should be squashed with patch 16.
Ideas for eliminating the code duplication?
vcs-svn/svndiff.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++++
1 files changed, 59 insertions(+), 0 deletions(-)
create mode 100644 vcs-svn/svndiff.c
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:45
The magic number of svn deltas is SVN followed by a null byte.
An alternative format (with compressed text) uses magic number SVN\1,
but that is deprecated in favor of compressing the deltas as a whole
as far as I can tell.
Signed-off-by: Jonathan Nieder <redacted>
---
vcs-svn/svndiff.c | 18 ++++++++++++++++++
1 files changed, 18 insertions(+), 0 deletions(-)
@@ -20,6 +21,23 @@#define VLI_DIGIT_MASK 0x7f#define VLI_BITS_PER_DIGIT 7+staticintread_magic(structline_buffer*in,off_t*len)+{+staticconstcharmagic[]={'S','V','N','\0'};+structstrbufsb=STRBUF_INIT;+if(*len<sizeof(magic))+returnerror("Invalid delta: no file type header");+buffer_read_binary(&sb,sizeof(magic),in);+if(sb.len!=sizeof(magic))+returnerror("Invalid delta: no file type header");+if(memcmp(sb.buf,magic,sizeof(magic)))+returnerror("Unrecognized file type %.*s",+(int)sizeof(magic),sb.buf);+*len-=sizeof(magic);+strbuf_release(&sb);+return0;+}+staticintread_int(structline_buffer*in,uintmax_t*result,off_t*len){off_tsz=*len;
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:45
The idiom (a + b < a) works fine for detecting that an unsigned
integer has overflowed, but the more explicit
unsigned_add_overflows(a, b)
might be easier to read.
Define such a macro, expanding roughly to ((a) < UINT_MAX - (b)).
Because the expansion uses each argument only once outside of sizeof()
expressions, it is safe to use this macro with arguments that have
side-effects.
Signed-off-by: Jonathan Nieder <redacted>
---
git-compat-util.h | 6 ++++++
1 files changed, 6 insertions(+), 0 deletions(-)
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:45
From: Ramkumar Ramachandra <redacted>
The svn-fe test script only requires git and the svn command-line
tools. Make these tests easier to read and run by not using the perl
libsvn bindings and instead duplicating only the relevant code from
lib-git-svn.sh.
Signed-off-by: Ramkumar Ramachandra <redacted>
Signed-off-by: Jonathan Nieder <redacted>
---
t/t9010-svn-fe.sh | 14 ++++++++++++--
1 files changed, 12 insertions(+), 2 deletions(-)
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:45
The buffer_read_string() function returns a temporary string of
size specified by the caller. It currently uses an obj_pool to
store the return value, but that is overkill: all we need is a
buffer that can grow between requests to accomodate larger
strings.
Use a strbuf instead.
Signed-off-by: Jonathan Nieder <redacted>
---
[Resent after messing up the message header; sorry for the noise.]
vcs-svn/line_buffer.c | 16 ++++++----------
1 files changed, 6 insertions(+), 10 deletions(-)
@@ -5,15 +5,13 @@#include"git-compat-util.h"#include"line_buffer.h"-#include"obj_pool.h"+#include"strbuf.h"#define LINE_BUFFER_LEN 10000#define COPY_BUFFER_LEN 4096-/* Create memory pool for char sequence of known length */-obj_pool_gen(blob,char,4096)-staticcharline_buffer[LINE_BUFFER_LEN];+staticstructstrbufblob_buffer=STRBUF_INIT;staticFILE*infile;intbuffer_init(constchar*filename)
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:45
Implement an svndiff 0 interpreter, for use by the dumpfilev3
importer. It is slower than it needs to be (e.g., it does not use
fseek() on input) for simplicity.
This is based only on the spec and not Subversion's implementation of
the svndiff0 spec.
The tests come from various deltas encountered in importing the
Apache SVN repo.
The svndiff0 semantics are not completely documented, meaning that
some of this work had to be done by guesswork. It is not complete.
This version of the patch omits the enormous Xerces.cpp test (which
is not so interesting because it passes, anyway).
Helped-by: Ramkumar Ramachandra [off-list ref]
Helped-by: Thomas Rast [off-list ref]
Helped-by: David Barr [off-list ref]
Not-signed-off-by: Jonathan Nieder [off-list ref]
---
[alternate ending, for those who do not like reading 3-MiB messages -
sorry about that]
Still not signed off because I haven't checked the copyright of the
tests. I would prefer tiny deltas or deltas of some public-domain
work (e.g., drafts of foundation documents of some country).
This patch owes a great deal to David and Ram, probably more than
it owes to me. David made lots and lots of fixes. Ram introduced
the tests and helped with design. Thomas provided an early sanity
check for code clarity.
Thanks for reading.
Makefile | 4 +-
t/t9010-svn-fe.sh | 14 ++
t/t9010/newdata.diff0 | Bin 0 -> 19392 bytes
t/t9010/newdata.done | 522 +++++++++++++++++++++++++++++++++++++++++++++++++
t/t9010/src.diff0 | Bin 0 -> 74 bytes
t/t9010/src.done | 522 +++++++++++++++++++++++++++++++++++++++++++++++++
test-svn-fe.c | 37 +++-
vcs-svn/svndiff.c | 265 +++++++++++++++++++++++++
vcs-svn/svndiff.h | 9 +
9 files changed, 1364 insertions(+), 9 deletions(-)
create mode 100644 t/t9010/blank.done
create mode 100644 t/t9010/newdata.diff0
create mode 100644 t/t9010/newdata.done
create mode 100644 t/t9010/src.diff0
create mode 100644 t/t9010/src.done
create mode 100644 vcs-svn/svndiff.h
@@ -0,0 +1,522 @@+APACHE COMMONS PROJECT+STATUS: -*-indented-text-*- Last modified at [$Date$]++Background:+ o IRC channel #apache-commons on irc.openprojects.net+ traffic is logged to <URL:http://Source-Zone.Org/apache-irc/>+ so that the content of interactive discussions is available+ to everyone++Project committers (as of 2002-10-27):+ o commons:+ aaron,coar,donaldp,jerenkrantz,fitz,geirm,gstein,jim,striker+ o commons-site:+ aaron,coar,donaldp,jerenkrantz,fitz,geirm,gstein,jim,striker,+ sanders,nicolaken++Release:+ none yet; still defining mission :-)+++Resolved Issues:++ o Commons is a parent of reusable code projects. These projects+ may be used by other projects of the ASF, but it is not a+ requirement.++ o The Commons will be language-agnostic.++ o Projects that are "in scope" are defined as:++ - Existing components that are, or would be, useful to multiple+ projects++ - If a component does not fit the (TBD) goals of Apache Commons,+ then it is not considered "in scope" just because it has no+ other home. In other words, the Apache Commons is not a place+ of last refuge if the component does not match the Apache+ Commons' goals.++ - Reusable libraries+ [ gstein: we should expand this definition for the mission+ statement; examples provided were serf and regexp ]++ - Components that do not fit cleanly into any other top-level+ project, but they do fit the goals of Commons.++ o Voting will follow the "standard Apache voting guidelines"++ [ be nice to refer to an Incubator doc here ]++ o All code donations [to the ASF, destined for Apache Commons]+ arrive via the Incubator, unless the Incubator states they can+ be placed directly into Commons.++ o Existing Commons committers can start new components without a+ detour to the Incubator. These new components must be approved+ by the PMC and must meet the (TBD) goals of Apache Commons.+++Pending issues:+ o Coming up with a set of bylaws for the project++ o Enabling Reply-to on the @commons lists+ (pmc@ will *not* use reply-to munging, but user lists+ will be determined by user majority; this item applies+ to lists for which the decision has not yet been made)+ +1: aaron, coar, donaldp, geirm, acoliver, mas, bayard, sanders+ -1: fitz, gstein, jerenkrantz, striker, jim++ o The name 'Commons' has caused some heartburn with the+ Jakarta community because of the Jakarta-Commons project.+ Should we rename to avoid conflicts and keep the peace?+ Conflicts would include Java namespace as well as+ philosophical aspects.+ +1: + +0: coar (i'm willing)+ -0: jerenkrantz, donaldp, striker, gstein, fitz+ -1: sanders++ o If we rename, to what? What words/names describe our+ purpose?+ - toolbox+ +0: gstein (I'd be +1 but for the confusion with the existing+ Apache Toolbox project, but *really* like this+ name)+ - toolchest+ +0.5: gstein+ - tools+ +0: gstein+ -1: donaldp (tools are different to components)+ - components+ +0: gstein (a bit long)+ - util+ - library+ +0: gstein (doesn't fit well with perl/python "modules")+ - suite (sweet?)+ - belt (as in bat-belt or tool-belt)+ - mcgyver+ - foundry or mill+ +1: sanders (maybe too 'SourceForgeesque')+ -0: donaldp (If reorg goes through we may have multiple+ foundaries or federations for different "concepts")+ - federation+ - share or shared+ - stuff+ +.3: fitz :)+ - ?++ o Style for the mailing lists:++ One community mailing list, with specific breakouts:+ +1: fitz, jerenkrantz, sanders, coar,+ donaldp (lets start here and evolve)+ +0.5: mas+ -0: aaron (too early)++ Topical mailing lists:+ +1: gstein, scolebourne, acoliver, striker+ -0: aaron (too early), jerenkrantz+ -0.1: mas, sanders (too early for this), donaldp+ -1: coar++ Per-language mailing lists:+ -0: aaron (too early)+ -0.1: mas+ -1: gstein, sanders, fitz, jerenkrantz, striker, coar++ Per-component mailing lists as a default (breakouts will create+ these as a matter of course, this is about the default)+ +0.7: mas+ -0: aaron (too early)+ -0.9: sanders+ -1: gstein, fitz, jerenkrantz, striker++ o A number of very valid issues have been brought up on the+ list. We need to figure out how the Commons Project will+ deal with each of these, in terms of new components and+ how those components will contain code projects. This list+ is only meant to keep record of all the issues:++ - Releasable pieces+ - Release rules+ - Voting scope+ - Directory structure and naming conventions+ - Coding style+ - Build system consistency (or inconsistency)+ - Namespace issues (esp. w/ java)+ - Language vs. Functional++ o Default commit privileges++ - Commons-wide+ +1:+ -1: gstein, striker, donaldp++ - Per-component+ +1: gstein, striker, donaldp, jerenkrantz+ -1:++ - Per-component with self-chosen aggregation+ +1: gstein, donaldp+ -1:++ o Granularity of CVS repositories for components (this excludes+ commons-site)++ - Commons-wide+ +1: gstein, donaldp, jerenkrantz+ -1:++ - Per-topic+ +1:+ -0: gstein, donaldp, jerenkrantz+ -1: ++ - Per-component+ +1:+ -1: gstein, donaldp, jerenkrantz+++Project Mission:++What is the project's mission? Our statement of goals/mission/vision+should arise from the answers to the following and other questions:+(jim notes that defining something after the fact seems very backwards+ and broken; gstein notes that we're refining the board-provided+ charter)++ o Should commons have an sandbox component to ease infrastructure+ burden on smaller code bases?+ +1: coar, donaldp, jerenkrantz, gstein, sanders (non-binding)+ +0: fitz+ -0: striker+ -1: jim (the PMC is about reusability, not sandbox),+ aaron (what jim said; and go see incubator)++ o What types of components would be appropriate for this project? + ("in scope")++ - Tools that help/promote reusability?+ Hypothetical: ant, jlibtool, ASF-based autoconf+ +1: jerenkrantz, gstein, striker, fitz, sanders (non-binding)+ -0: donaldp (prefer a tools PMC for that)+ -1: aaron (too broad, don't belong here)++ - Development frameworks?+ Hypothetical: avalon+ +1: fitz+ -0: donaldp (how do we determine this given we would prolly+ accept it if it was new?)+ -1: gstein (the avalon components, but not the whole bugger),+ striker, sanders (non-binding)++ - Components that fit the (TBD) goals of Commons, have a more+ "logical" home elsewhere in the ASF, but were rejected by that+ home?+ +1: gstein, donaldp+ 0: striker (on a case by case basis, taking reasons for rejection+ seriously into account. Abstain from vote until+ rephrased),+ fitz (what striker said), aaron (what fitz said)+ -1: jerenkrantz, sanders (non-binding)++ FOLD BELOW VOTES INTO ABOVE? (i.e. eliminate the "donation" wording)+ - Donations that could fit but have a more obvious (proper) home which+ has already rejected it?+ +1: coar, donaldp, gstein (note the "might fit" term)+ -0:+ -1: jerenkrantz, jim, aaron, striker, fitz++ - Existing ASF components whose committers believe that they+ are a better fit under commons and the commons PMC agrees?+ (If this component were brought up as new, we would accept it.)+ +1: coar, donaldp, jerenkrantz, striker, gstein, fitz+ -1: jim (by this definition httpd could be in commons)+ (gstein says: see the "if" part; we wouldn't accept httpd)+ (jim says: until we better define what the PMC would or+ would not accept, then this seems too wishy-washy to me)+ (gstein says: jim, you're blocking closure on this;+ how would you refine the phrasing here; the intent+ here is to accept components from the other Commons+ projects or projects with reusable component),+ aaron (we need to differenciate ourselves from other+ libraries first, namely APR)++ - Packages being worked on by Apache developers, with a clear+ affiliation, that can't or won't be bundled? (E.g., an+ httpd module)+ +1: coar, donaldp+ -1: jerenkrantz, striker, gstein, fitz, jim, aaron+ CLOSE THIS? (as "not passed"; what is a good way to phrase this?)++ - Should we have a minimum bar of entry for components?+ +1:+ -0: donaldp, gstein+ -1:++ - Should we have a minimum set of requirements before components+ are released?+ +1: donaldp, gstein (mixed, see below), striker+ -1: jerenkrantz (what is released?)++ - If yes to above then which things should be part of minimum+ requirements?++ documentation: require basic overview and user docs+ +1: donaldp+ -0: gstein (recommend highly, but let the committers determine+ what is right for the component),+ striker, jerenkrantz+ -1:++ uptodate website: require website be updated to latest release+ but may still host previous release docs.+ +1: donaldp, gstein, striker+ -0: jerenkrantz+ -1:++ unit tests: (okay so this will never get consensus but ...)+ +1: donaldp+ -1: gstein (unit tests should be recommended, but not+ mandated; I also find it unreasonable for initial+ development/pre-alpha releases, but it can make+ sense for "final" types of releases),+ striker, jerenkrantz++ versioning standard: derived from+ http://apr.apache.org/versioning.html+ http://jakarta.apache.org/commons/versioning.html+ +1: donaldp, gstein, striker, jerenkrantz+ -1:++ release process: derived from+ http://jakarta.apache.org/commons/releases.html+ http://jakarta.apache.org/turbine/maven/development/release-process.html+ http://cvs.apache.org/viewcvs.cgi/jakarta-ant/ReleaseInstructions?rev=1.9.2.1&content-type=text/vnd.viewcvs-markup+ +1: donaldp+ -1: gstein (we should provide "best practices" but allow each+ components' committers to define their rules),+ striker, jerenkrantz++ deprecation process: (java specific?)+ http://jakarta.apache.org/turbine/maven/development/deprecation.html+ +1: donaldp, gstein (I see this as part of the "versioning"+ process, and we can provide best+ practices here)+ -0: jerenkrantz (kinda sorta versioning, but not quite)+ -1:++ CVS/Subversion branching:+ http://jakarta.apache.org/turbine/maven/development/branches.html+ +1: donaldp+ -1: gstein (we should provide "best practices" but allow each+ components' committers to define their rules),+ striker, jerenkrantz+++Candidate Projects:++ o APR's serf project has voted itself to move into Commons.++ - Should the PMC accept it as fitting the Commons goal?+ +1: gstein, fitz, jerenkrantz, striker, donaldp+ -1: aaron (no such thing as "the Commons goal", how can it fit it?)++ - When should it move?++ Whenever it likes:+ +1: gstein, sanders, jerenkrantz, striker+ +0: donaldp (+1 if we use subversion, but if using CVS + we should hold off until structure is decided upon)+ -1: aaron (after we know why it fits)++ Give us a while:+ +1: fitz (what's the hurry?), aaron+ -0: gstein (we're only talking about a small seed of a+ codebase; it won't get in our way as we complete the+ charter), striker++ - Where should the CVS code be located?++ commons/serf (each component under top-level)+ +1: sanders (works well at jakarta-commons)+ fitz (Please don't mix interface and implementation + of commons!), aaron+ +0: jerenkrantz+ -0.5: gstein+ -1: donaldp (makes it difficult to update all related + projects with a single sweep)++ commons/components/serf (all components under this dir,+ leaving the top open for other non-code items)+ +0: gstein, striker, donaldp (is this just dev with a + different name?+ (gstein says "yes"))+ -1: fitz, aaron, jerenkrantz++ commons/clients/serf (topical-groups under top-level)+ +1: gstein, jerenkrantz+ -1: fitz, aaron, donaldp++ commons/dev/serf (all components under "dev")+ +1: gstein, donaldp (if we are having a single + monolithic repo for all commons)+ -1: fitz, aaron, jerenkrantz++ commons/bootstrap/serf (serf is very early stage, so maybe we+ have a "bootstrap" area; this is different from Incubator+ since the existing committers do not need "training")+ +1: gstein, donaldp+ -1: fitz, aaron, jerenkrantz++ commons/???++ commons/c/serf (separate out component based on language+ and then have a flat structure underneath)+ +1: donaldp+ -1: jerenkrantz++ - What mailing list should it use for dev discussions?++ general@commons.apache.org: (one group for all discussion;+ dev and non-dev alike)+ -0.5: gstein+ -1: striker, aaron, jerenkrantz++ dev@commons.apache.org: (one group for dev discussion;+ general@ remains for non-dev)+ +1: gstein, fitz, sanders, jerenkrantz, striker, donaldp++ clients-dev@commons.apache.org:+ (this is really TOPICNAME-dev@ where I preselected+ "clients" for TOPICNAME; this question is whether this+ style would be appropriate)+ +1: gstein, striker+ -0: sanders, donaldp (maybe in the future but too early),+ jerenkrantz+ -1: aaron (what is "clients"? I'd probably be +1 if I knew+ what that was)++ - Note: serf has no web site, so there isn't a need to figure+ that out right now.+++Assets:+ DNS: commons.apache.org++ Mailing lists: general@commons.apache.org+ announce@commons.apache.org+ pmc@commons.apache.org+ cvs@commons.apache.org++ [ core-cvs@commons.apache.org in case we+ create a commons-core CVS module ]++ Web site: http://commons.apache.org/++ Repositories: commons (code, info, etc)+ commons-site (the web site)+++PMC Members:++ Aaron Bannert <aaron@apache.org>+ Ken Coar <coar@apache.org>+ Peter Donald <peter@apache.org>+ Justin Erenkrantz <jerenkrantz@apache.org>+ Brian W. Fitzpatrick <fitz@apache.org>+ Jim Jagielski <jim@apache.org>+ Geir Magnusson Jr. <geirm@apache.org>+ Greg Stein <gstein@lyra.org>+ Sander Striker <striker@apache.org>++ Note: Ken Coar is the Chair+++PMC Members, pending Board approval:++ none yet++ [ this may become obsolete; the Board is discussing a way for the+ Chair to directly alter the PMC membership; until then, however,+ we need PMC members ratified by the board, and this tracks them ]+++Committers:++ none yet [still defining mission]+++Invited Committers:++ none yet+++Current mission/charter as approved by the board:++ 'The Apache Commons PMC hereby is responsible for the creation+ and maintenance of software related to reusable libraries and+ components, based on software licensed to the Foundation.'++The complete text of the resolution that was passed is:++ WHEREAS, the Board of Directors deems it to be in the best+ interests of the Foundation and consistent with the+ Foundation's purpose to establish a Project Management+ Committee charged with the creation and maintenance of+ open-source software related to reusable libraries and+ components, for distribution at no charge to the public.++ NOW, THEREFORE, BE IT RESOLVED, that a Project Management+ Committee (PMC), to be known as the "Apache Commons PMC", be+ and hereby is established pursuant to Bylaws of the Foundation;+ and be it further++ RESOLVED, that the Apache Commons PMC be and hereby is+ responsible for the creation and maintenance of software+ related to reusable libraries and components, based on software+ licensed to the Foundation; and be it further++ RESOLVED, that the office of "Vice President, Apache Commons"+ be and hereby is created, the person holding such office to+ serve at the direction of the Board of Directors as the chair+ of the Apache Commons PMC, and to have primary responsibility+ for management of the projects within the scope of+ responsibility of the Apache Commons PMC; and be it further++ RESOLVED, that the persons listed immediately below be and+ hereby are appointed to serve as the initial members of the+ Apache Commons PMC:++ Aaron Bannert+ Ken Coar (chair)+ Peter Donald+ Justin Erenkrantz+ Brian W. Fitzpatrick+ Jim Jagielski+ Geir Magnusson Jr.+ Greg Stein+ Sander Striker++ NOW, THEREFORE, BE IT FURTHER RESOLVED, that Ken Coar be and+ hereby is appointed to the office of Vice President, Apache+ Commons, to serve in accordance with and subject to the+ direction of the Board of Directors and the Bylaws of the+ Foundation until death, resignation, retirement, removal or+ disqualification, or until a successor is appointed; and be it+ further++ RESOLVED, that the initial Apache Commons PMC be and hereby is+ tasked with the creation of a set of bylaws intended to+ encourage open development and increased participation in the+ Apache Commons Project.++#+# Local Variables:+# mode: indented-text+# tab-width: 4+# indent-tabs-mode: nil+# tab-stop-list: (4 6 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80)+# End:+#
@@ -0,0 +1,522 @@+APACHE COMMONS PROJECT+STATUS: -*-indented-text-*- Last modified at [$Date$]++Background:+ o IRC channel #apache-commons on irc.openprojects.net+ traffic is logged to <URL:http://Source-Zone.Org/apache-irc/>+ so that the content of interactive discussions is available+ to everyone++Project committers (as of 2002-10-27):+ o commons:+ aaron,coar,donaldp,jerenkrantz,fitz,geirm,gstein,jim,striker+ o commons-site:+ aaron,coar,donaldp,jerenkrantz,fitz,geirm,gstein,jim,striker,+ sanders,nicolaken++Release:+ none yet; still defining mission :-)+++Resolved Issues:++ o Commons is a parent of reusable code projects. These projects+ may be used by other projects of the ASF, but it is not a+ requirement.++ o The Commons will be language-agnostic.++ o Projects that are "in scope" are defined as:++ - Existing components that are, or would be, useful to multiple+ projects++ - If a component does not fit the (TBD) goals of Apache Commons,+ then it is not considered "in scope" just because it has no+ other home. In other words, the Apache Commons is not a place+ of last refuge if the component does not match the Apache+ Commons' goals.++ - Reusable libraries+ [ gstein: we should expand this definition for the mission+ statement; examples provided were serf and regexp ]++ - Components that do not fit cleanly into any other top-level+ project, but they do fit the goals of Commons.++ o Voting will follow the "standard Apache voting guidelines"++ [ be nice to refer to an Incubator doc here ]++ o All code donations [to the ASF, destined for Apache Commons]+ arrive via the Incubator, unless the Incubator states they can+ be placed directly into Commons.++ o Existing Commons committers can start new components without a+ detour to the Incubator. These new components must be approved+ by the PMC and must meet the (TBD) goals of Apache Commons.+++Pending issues:+ o Co o Subversion will be used for version controlComing up with a set of bylaws for the project++ o Enabling Reply-to on the @commons lists+ (pmc@ will *not* use reply-to munging, but user lists+ will be determined by user majority; this item applies+ to lists for which the decision has not yet been made)+ +1: aaron, coar, donaldp, geirm, acoliver, mas, bayard, sanders+ -1: fitz, gstein, jerenkrantz, striker, jim++ o The name 'Commons' has caused some heartburn with the+ Jakarta community because of the Jakarta-Commons project.+ Should we rename to avoid conflicts and keep the peace?+ Conflicts would include Java namespace as well as+ philosophical aspects.+ +1: + +0: coar (i'm willing)+ -0: jerenkrantz, donaldp, striker, gstein, fitz+ -1: sanders++ o If we rename, to what? What words/names describe our+ purpose?+ - toolbox+ +0: gstein (I'd be +1 but for the confusion with the existing+ Apache Toolbox project, but *really* like this+ name)+ - toolchest+ +0.5: gstein+ - tools+ +0: gstein+ -1: donaldp (tools are different to components)+ - components+ +0: gstein (a bit long)+ - util+ - library+ +0: gstein (doesn't fit well with perl/python "modules")+ - suite (sweet?)+ - belt (as in bat-belt or tool-belt)+ - mcgyver+ - foundry or mill+ +1: sanders (maybe too 'SourceForgeesque')+ -0: donaldp (If reorg goes through we may have multiple+ foundaries or federations for different "concepts")+ - federation+ - share or shared+ - stuff+ +.3: fitz :)+ - ?++ o Style for the mailing lists:++ One community mailing list, with specific breakouts:+ +1: fitz, jerenkrantz, sanders, coar,+ donaldp (lets start here and evolve)+ +0.5: mas+ -0: aaron (too early)++ Topical mailing lists:+ +1: gstein, scolebourne, acoliver, striker+ -0: aaron (too early), jerenkrantz+ -0.1: mas, sanders (too early for this), donaldp+ -1: coar++ Per-language mailing lists:+ -0: aaron (too early)+ -0.1: mas+ -1: gstein, sanders, fitz, jerenkrantz, striker, coar++ Per-component mailing lists as a default (breakouts will create+ these as a matter of course, this is about the default)+ +0.7: mas+ -0: aaron (too early)+ -0.9: sanders+ -1: gstein, fitz, jerenkrantz, striker++ o A number of very valid issues have been brought up on the+ list. We need to figure out how the Commons Project will+ deal with each of these, in terms of new components and+ how those components will contain code projects. This list+ is only meant to keep record of all the issues:++ - Releasable pieces+ - Release rules+ - Voting scope+ - Directory structure and naming conventions+ - Coding style+ - Build system consistency (or inconsistency)+ - Namespace issues (esp. w/ java)+ - Language vs. Functional++ o Default commit privileges++ - Commons-wide+ +1:+ -1: gstein, striker, donaldp++ - Per-component+ +1: gstein, striker, donaldp, jerenkrantz+ -1:++ - Per-component with self-chosen aggregation+ +1: gstein, donaldp+ -1:++ o Granularity of CVS repositories for components (this excludes+ commons-site)++ - Commons-wide+ +1: gstein, donaldp, jerenkrantz+ -1:++ - Per-topic+ +1:+ -0: gstein, donaldp, jerenkrantz+ -1: ++ - Per-component+ +1:+ -1: gstein, donaldp, jerenkrantz+++Project Mission:++What is the project's mission? Our statement of goals/mission/vision+should arise from the answers to the following and other questions:+(jim notes that defining something after the fact seems very backwards+ and broken; gstein notes that we're refining the board-provided+ charter)++ o Should commons have an sandbox component to ease infrastructure+ burden on smaller code bases?+ +1: coar, donaldp, jerenkrantz, gstein, sanders (non-binding)+ +0: fitz+ -0: striker+ -1: jim (the PMC is about reusability, not sandbox),+ aaron (what jim said; and go see incubator)++ o What types of components would be appropriate for this project? + ("in scope")++ - Tools that help/promote reusability?+ Hypothetical: ant, jlibtool, ASF-based autoconf+ +1: jerenkrantz, gstein, striker, fitz, sanders (non-binding)+ -0: donaldp (prefer a tools PMC for that)+ -1: aaron (too broad, don't belong here)++ - Development frameworks?+ Hypothetical: avalon+ +1: fitz+ -0: donaldp (how do we determine this given we would prolly+ accept it if it was new?)+ -1: gstein (the avalon components, but not the whole bugger),+ striker, sanders (non-binding)++ - Components that fit the (TBD) goals of Commons, have a more+ "logical" home elsewhere in the ASF, but were rejected by that+ home?+ +1: gstein, donaldp+ 0: striker (on a case by case basis, taking reasons for rejection+ seriously into account. Abstain from vote until+ rephrased),+ fitz (what striker said), aaron (what fitz said)+ -1: jerenkrantz, sanders (non-binding)++ FOLD BELOW VOTES INTO ABOVE? (i.e. eliminate the "donation" wording)+ - Donations that could fit but have a more obvious (proper) home which+ has already rejected it?+ +1: coar, donaldp, gstein (note the "might fit" term)+ -0:+ -1: jerenkrantz, jim, aaron, striker, fitz++ - Existing ASF components whose committers believe that they+ are a better fit under commons and the commons PMC agrees?+ (If this component were brought up as new, we would accept it.)+ +1: coar, donaldp, jerenkrantz, striker, gstein, fitz+ -1: jim (by this definition httpd could be in commons)+ (gstein says: see the "if" part; we wouldn't accept httpd)+ (jim says: until we better define what the PMC would or+ would not accept, then this seems too wishy-washy to me)+ (gstein says: jim, you're blocking closure on this;+ how would you refine the phrasing here; the intent+ here is to accept components from the other Commons+ projects or projects with reusable component),+ aaron (we need to differenciate ourselves from other+ libraries first, namely APR)++ - Packages being worked on by Apache developers, with a clear+ affiliation, that can't or won't be bundled? (E.g., an+ httpd module)+ +1: coar, donaldp+ -1: jerenkrantz, striker, gstein, fitz, jim, aaron+ CLOSE THIS? (as "not passed"; what is a good way to phrase this?)++ - Should we have a minimum bar of entry for components?+ +1:+ -0: donaldp, gstein+ -1:++ - Should we have a minimum set of requirements before components+ are released?+ +1: donaldp, gstein (mixed, see below), striker+ -1: jerenkrantz (what is released?)++ - If yes to above then which things should be part of minimum+ requirements?++ documentation: require basic overview and user docs+ +1: donaldp+ -0: gstein (recommend highly, but let the committers determine+ what is right for the component),+ striker, jerenkrantz+ -1:++ uptodate website: require website be updated to latest release+ but may still host previous release docs.+ +1: donaldp, gstein, striker+ -0: jerenkrantz+ -1:++ unit tests: (okay so this will never get consensus but ...)+ +1: donaldp+ -1: gstein (unit tests should be recommended, but not+ mandated; I also find it unreasonable for initial+ development/pre-alpha releases, but it can make+ sense for "final" types of releases),+ striker, jerenkrantz++ versioning standard: derived from+ http://apr.apache.org/versioning.html+ http://jakarta.apache.org/commons/versioning.html+ +1: donaldp, gstein, striker, jerenkrantz+ -1:++ release process: derived from+ http://jakarta.apache.org/commons/releases.html+ http://jakarta.apache.org/turbine/maven/development/release-process.html+ http://cvs.apache.org/viewcvs.cgi/jakarta-ant/ReleaseInstructions?rev=1.9.2.1&content-type=text/vnd.viewcvs-markup+ +1: donaldp+ -1: gstein (we should provide "best practices" but allow each+ components' committers to define their rules),+ striker, jerenkrantz++ deprecation process: (java specific?)+ http://jakarta.apache.org/turbine/maven/development/deprecation.html+ +1: donaldp, gstein (I see this as part of the "versioning"+ process, and we can provide best+ practices here)+ -0: jerenkrantz (kinda sorta versioning, but not quite)+ -1:++ CVS/Subversion branching:+ http://jakarta.apache.org/turbine/maven/development/branches.html+ +1: donaldp+ -1: gstein (we should provide "best practices" but allow each+ components' committers to define their rules),+ striker, jerenkrantz+++Candidate Projects:++ o APR's serf project has voted itself to move into Commons.++ - Should the PMC accept it as fitting the Commons goal?+ +1: gstein, fitz, jerenkrantz, striker, donaldp+ -1: aaron (no such thing as "the Commons goal", how can it fit it?)++ - When should it move?++ Whenever it likes:+ +1: gstein, sanders, jerenkrantz, striker+ +0: donaldp (+1 if we use subversion, but if using CVS + we should hold off until structure is decided upon)+ -1: aaron (after we know why it fits)++ Give us a while:+ +1: fitz (what's the hurry?), aaron+ -0: gstein (we're only talking about a small seed of a+ codebase; it won't get in our way as we complete the+ charter), striker++ - Where should the CVS code be located?++ commons/serf (each component under top-level)+ +1: sanders (works well at jakarta-commons)+ fitz (Please don't mix interface and implementation + of commons!), aaron+ +0: jerenkrantz+ -0.5: gstein+ -1: donaldp (makes it difficult to update all related + projects with a single sweep)++ commons/components/serf (all components under this dir,+ leaving the top open for other non-code items)+ +0: gstein, striker, donaldp (is this just dev with a + different name?+ (gstein says "yes"))+ -1: fitz, aaron, jerenkrantz++ commons/clients/serf (topical-groups under top-level)+ +1: gstein, jerenkrantz+ -1: fitz, aaron, donaldp++ commons/dev/serf (all components under "dev")+ +1: gstein, donaldp (if we are having a single + monolithic repo for all commons)+ -1: fitz, aaron, jerenkrantz++ commons/bootstrap/serf (serf is very early stage, so maybe we+ have a "bootstrap" area; this is different from Incubator+ since the existing committers do not need "training")+ +1: gstein, donaldp+ -1: fitz, aaron, jerenkrantz++ commons/???++ commons/c/serf (separate out component based on language+ and then have a flat structure underneath)+ +1: donaldp+ -1: jerenkrantz++ - What mailing list should it use for dev discussions?++ general@commons.apache.org: (one group for all discussion;+ dev and non-dev alike)+ -0.5: gstein+ -1: striker, aaron, jerenkrantz++ dev@commons.apache.org: (one group for dev discussion;+ general@ remains for non-dev)+ +1: gstein, fitz, sanders, jerenkrantz, striker, donaldp++ clients-dev@commons.apache.org:+ (this is really TOPICNAME-dev@ where I preselected+ "clients" for TOPICNAME; this question is whether this+ style would be appropriate)+ +1: gstein, striker+ -0: sanders, donaldp (maybe in the future but too early),+ jerenkrantz+ -1: aaron (what is "clients"? I'd probably be +1 if I knew+ what that was)++ - Note: serf has no web site, so there isn't a need to figure+ that out right now.+++Assets:+ DNS: commons.apache.org++ Mailing lists: general@commons.apache.org+ announce@commons.apache.org+ pmc@commons.apache.org+ cvs@commons.apache.org++ [ core-cvs@commons.apache.org in case we+ create a commons-core CVS module ]++ Web site: http://commons.apache.org/++ Repositories: commons (code, info, etc)+ commons-site (the web site)+++PMC Members:++ Aaron Bannert <aaron@apache.org>+ Ken Coar <coar@apache.org>+ Peter Donald <peter@apache.org>+ Justin Erenkrantz <jerenkrantz@apache.org>+ Brian W. Fitzpatrick <fitz@apache.org>+ Jim Jagielski <jim@apache.org>+ Geir Magnusson Jr. <geirm@apache.org>+ Greg Stein <gstein@lyra.org>+ Sander Striker <striker@apache.org>++ Note: Ken Coar is the Chair+++PMC Members, pending Board approval:++ none yet++ [ this may become obsolete; the Board is discussing a way for the+ Chair to directly alter the PMC membership; until then, however,+ we need PMC members ratified by the board, and this tracks them ]+++Committers:++ none yet [still defining mission]+++Invited Committers:++ none yet+++Current mission/charter as approved by the board:++ 'The Apache Commons PMC hereby is responsible for the creation+ and maintenance of software related to reusable libraries and+ components, based on software licensed to the Foundation.'++The complete text of the resolution that was passed is:++ WHEREAS, the Board of Directors deems it to be in the best+ interests of the Foundation and consistent with the+ Foundation's purpose to establish a Project Management+ Committee charged with the creation and maintenance of+ open-source software related to reusable libraries and+ components, for distribution at no charge to the public.++ NOW, THEREFORE, BE IT RESOLVED, that a Project Management+ Committee (PMC), to be known as the "Apache Commons PMC", be+ and hereby is established pursuant to Bylaws of the Foundation;+ and be it further++ RESOLVED, that the Apache Commons PMC be and hereby is+ responsible for the creation and maintenance of software+ related to reusable libraries and components, based on software+ licensed to the Foundation; and be it further++ RESOLVED, that the office of "Vice President, Apache Commons"+ be and hereby is created, the person holding such office to+ serve at the direction of the Board of Directors as the chair+ of the Apache Commons PMC, and to have primary responsibility+ for management of the projects within the scope of+ responsibility of the Apache Commons PMC; and be it further++ RESOLVED, that the persons listed immediately below be and+ hereby are appointed to serve as the initial members of the+ Apache Commons PMC:++ Aaron Bannert+ Ken Coar (chair)+ Peter Donald+ Justin Erenkrantz+ Brian W. Fitzpatrick+ Jim Jagielski+ Geir Magnusson Jr.+ Greg Stein+ Sander Striker++ NOW, THEREFORE, BE IT FURTHER RESOLVED, that Ken Coar be and+ hereby is appointed to the office of Vice President, Apache+ Commons, to serve in accordance with and subject to the+ direction of the Board of Directors and the Bylaws of the+ Foundation until death, resignation, retirement, removal or+ disqualification, or until a successor is appointed; and be it+ further++ RESOLVED, that the initial Apache Commons PMC be and hereby is+ tasked with the creation of a set of bylaws intended to+ encourage open development and increased participation in the+ Apache Commons Project.++#+# Local Variables:+# mode: indented-text+# tab-width: 4+# indent-tabs-mode: nil+# tab-stop-list: (4 6 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80)+# End:+#
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:46
Jonathan Nieder wrote:
Implement an svndiff 0 interpreter
I hear that was nigh unreadable, so here's a reroll. Less
cargo-cult support for broken deltas, more readability and tests.
Patches apply on top of "[PATCH 15/16] t9010 (svn-fe): Eliminate
dependency on svn perl bindings". As before, the end result
includes a 'test-svn-fe -d' command that can apply svndiff0-format
deltas, meaning less binary garbage to worry about as you puzzle
over that confusing "svnrdump dump" output in debugging sessions.
Questions? Improvements? Bugs?
Patch 1 is a fixup to the variable-length integer parsing code, to
report unexpected EOF (i.e., declared content length too long)
correctly when it occurs in the middle of such an integer.
Patch 2 is the svndiff0 interpreter in broad strokes: read window,
read window, read window, .... The patch doesn't encode any
knowledge about what actually goes _in_ a window aside from the
header, so it will error out for nonempty windows.
Patch 3 teaches the nacent interpreter to keep the appropriate
piece of the preimage in memory. This is probably earlier in the
series than it ought to be, but I wanted to try out the sliding
window code.
With patches 4 and 5, the interpreter learns to read the "data"
and "instructions" section of a window. The effect is observable
because it finds the beginning of the next window correctly.
Patch 6 is an example instruction (copyfrom_data).
Patches 7-8 introduce some sanity checks.
Patches 9 and 10 are another instruction (copyfrom_target) and
another sanity check.
Patch 11 is the last instruction (copyfrom_source). That's it.
You can apply deltas now!
If anything seems unclear, please don't spend time puzzling it
out --- just yell at me, so the code or documentation can be
cleaned up. Happy reading.
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:46
A delta in the subversion delta (svndiff0) format consists of the
magic bytes SVN\0 followed by a sequence of windows, each beginning
with a window header consisting of five integers (with variable-length
representation):
source view offset
source view length
output length
instructions length
auxiliary data length
Add an svndiff0_apply() function and test-svn-fe -d commandline tool
to parse such a delta in the special case of not including any
instructions or auxiliary data.
Later patches will add features to turn this into a fully functional
delta applier, for use by svn-fe in parsing the streams produced by
"svnrdump dump" and "svnadmin dump --deltas".
Helped-by: Ramkumar Ramachandra [off-list ref]
Helped-by: David Barr [off-list ref]
Signed-off-by: Jonathan Nieder <redacted>
---
Makefile | 4 +-
t/t9011-svn-da.sh | 82 +++++++++++++++++++++++++++++++++++++++++++++++++++++
test-svn-fe.c | 39 ++++++++++++++++++++----
vcs-svn/svndiff.c | 64 +++++++++++++++++++++++++++++++++++++++++
vcs-svn/svndiff.h | 9 ++++++
5 files changed, 189 insertions(+), 9 deletions(-)
create mode 100755 t/t9011-svn-da.sh
create mode 100644 vcs-svn/svndiff.h
@@ -4,14 +4,39 @@#include"git-compat-util.h"#include"vcs-svn/svndump.h"+#include"vcs-svn/svndiff.h"+#include"vcs-svn/line_buffer.h"intmain(intargc,char*argv[]){-if(argc!=2)-usage("test-svn-fe <file>");-svndump_init(argv[1]);-svndump_read(NULL);-svndump_deinit();-svndump_reset();-return0;+staticconstchartest_svnfe_usage[]=+"test-svn-fe (<dumpfile> | [-d] <preimage> <delta> <len>)";+if(argc<2)+usage(test_svnfe_usage);+if(argc==2){+svndump_init(argv[1]);+svndump_read(NULL);+svndump_deinit();+svndump_reset();+return0;+}+if(argc==5&&!strcmp(argv[1],"-d")){+structline_bufferpreimage=LINE_BUFFER_INIT;+structline_bufferdelta=LINE_BUFFER_INIT;+if(buffer_init(&preimage,argv[2]))+die_errno("cannot open preimage");+if(buffer_init(&delta,argv[3]))+die_errno("cannot open delta");+if(svndiff0_apply(&delta,(off_t)strtoull(argv[4],NULL,0),+&preimage,stdout))+return1;+if(buffer_deinit(&preimage))+die_errno("cannot close preimage");+if(buffer_deinit(&delta))+die_errno("cannot close delta");+buffer_reset(&preimage);+buffer_reset(&delta);+return0;+}+usage(test_svnfe_usage);}
@@ -76,3 +77,66 @@ static int parse_int(const char **buf, size_t *result, const char *end)returnerror("Invalid instruction: incomplete integer %"PRIu64,(uint64_t)rv);}++staticintread_offset(structline_buffer*in,off_t*result,off_t*len)+{+uintmax_tval;+if(read_int(in,&val,len))+return-1;+if(val>maximum_signed_value_of_type(off_t))+returnerror("Unrepresentable offset: %"PRIuMAX,val);+*result=val;+return0;+}++staticintread_length(structline_buffer*in,size_t*result,off_t*len)+{+uintmax_tval;+if(read_int(in,&val,len))+return-1;+if(val>SIZE_MAX)+returnerror("Unrepresentable length: %"PRIuMAX,val);+*result=val;+return0;+}++staticintapply_one_window(structline_buffer*delta,off_t*delta_len)+{+size_tout_len;+size_tinstructions_len;+size_tdata_len;+assert(delta_len);++/* "source view" offset and length already handled; */+if(read_length(delta,&out_len,delta_len)||+read_length(delta,&instructions_len,delta_len)||+read_length(delta,&data_len,delta_len))+return-1;+if(instructions_len>0)+returnerror("What do you think I am? A delta applier?");+if(data_len>0)+returnerror("No support for inline data yet");+return0;+}++intsvndiff0_apply(structline_buffer*delta,off_tdelta_len,+structline_buffer*preimage,FILE*postimage)+{+assert(delta&&preimage&&postimage);++if(read_magic(delta,&delta_len))+return-1;+while(delta_len>0){/* For each window: */+off_tpre_off;+size_tpre_len;+if(read_offset(delta,&pre_off,&delta_len)||+read_length(delta,&pre_len,&delta_len)||+apply_one_window(delta,&delta_len))+return-1;+if(delta_len&&buffer_at_eof(delta))+returnerror("Delta ends early! "+"(%"PRIu64" bytes remaining)",+(uint64_t)delta_len);+}+return0;+}
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:46
The source view offset heading each svndiff0 window represents a
number of bytes past the beginning of the preimage. Together with the
source view length, it instructs the delta applier about what portion
of the preimage instructions will refer to. Read in that data right
away using the sliding window code.
Maybe some day we will mmap() to prepare to read data more lazily.
For compatibility with Subversion's implementation, tolerate source
view offsets pointing past the end of the preimage file (a later
patch will remove this flexibility). For simplicity, also permit
source views that start within the preimage and end outside of it,
even though Subversion does not.
This does not teach the delta applier to read instructions or copy
data from the source view yet. Deltas that would produce nonempty
output are still rejected.
Helped-by: Ramkumar Ramachandra [off-list ref]
Helped-by: David Barr [off-list ref]
Signed-off-by: Jonathan Nieder <redacted>
---
It occurs to me that Sam Vilain may well have something valuable to
say about this series, having implemented something similar[1].
Sam, this series adds an svndiff0 parser for git to use in parsing
v3 dumps (which are way easier to produce with remote access to an
svn repository than v2 dumps). The beginning of the series is at [2],
though that cover letter is out of date: now, modulo any new bugs
I've introduced with this reroll, it is known to successfully apply
all the deltas involved in a complete dump of the ASF repo.
I am interested in improvements and complaints of all kinds.
[1] http://search.cpan.org/~samv/Parse-SVNDiff-0.03/lib/Parse/SVNDiff.pm
[2] http://thread.gmane.org/gmane.comp.version-control.git/151086/focus=158731
t/t9011-svn-da.sh | 38 ++++++++++++++++++++++++++++++++++++++
vcs-svn/svndiff.c | 22 +++++++++++++++-------
2 files changed, 53 insertions(+), 7 deletions(-)
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:46
Each window of an svndiff0-format delta includes a section for new
data that will be copied into the preimage (in the order it appears in
the window, possibly interspersed with other data).
Read this data when encountering it. It is not actually necessary to
do so --- it would be just as easy to copy straight from the delta
to output when interpreting the relevant instructions --- but this
way, the code that interprets svndiff0 instructions can proceed more
quickly because it does not require any I/O.
Subversion's implementation rejects deltas that do not consume all
the auxiliary data that is available. Do not check that for now,
because it would make it impossible to test the function of this
patch until the instructions to consume data are implemented.
Do check for truncated data sections. Since Subversion's applier
rejects deltas that end before the new-data section is declared to
end, it should be safe for this applier to reject such deltas, too.
Helped-by: Ramkumar Ramachandra [off-list ref]
Helped-by: David Barr [off-list ref]
Signed-off-by: Jonathan Nieder <redacted>
---
t/t9011-svn-da.sh | 12 ++++++++++++
vcs-svn/svndiff.c | 27 ++++++++++++++++++++++++---
2 files changed, 36 insertions(+), 3 deletions(-)
@@ -115,9 +133,12 @@ static int apply_one_window(struct line_buffer *delta, off_t *delta_len)return-1;if(instructions_len>0)returnerror("What do you think I am? A delta applier?");-if(data_len>0)-returnerror("No support for inline data yet");-return0;+if(read_chunk(delta,delta_len,&ctx.data,data_len))+returnerror("Invalid delta: incomplete data section");+if(buffer_ferror(delta))+rv=error("Cannot read delta: %s",strerror(errno));+strbuf_release(&ctx.data);+returnrv;}intsvndiff0_apply(structline_buffer*delta,off_tdelta_len,
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:46
Buffer the instruction section upon encountering it for later
interpretation.
An alternative design would involve parsing the instructions
at this point and buffering them in some processed form. Using
the unprocessed form is simpler.
Signed-off-by: Jonathan Nieder <redacted>
---
t/t9011-svn-da.sh | 5 +++++
vcs-svn/svndiff.c | 23 ++++++++++++++++++-----
2 files changed, 23 insertions(+), 5 deletions(-)
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:46
The copyfrom_data instruction copies a few bytes verbatim from the
auxiliary data section of a window to the postimage.
Signed-off-by: Jonathan Nieder <redacted>
---
t/t9011-svn-da.sh | 31 +++++++++++++++++++
vcs-svn/svndiff.c | 86 +++++++++++++++++++++++++++++++++++++++++++++++++---
2 files changed, 112 insertions(+), 5 deletions(-)
@@ -146,8 +220,10 @@ static int apply_one_window(struct line_buffer *delta, off_t *delta_len)rv=error("Cannot read delta: %s",strerror(errno));gotodone;}-if(instructions_len>0)-returnerror("What do you think I am? A delta applier?");+if(apply_window_in_core(&ctx)||write_strbuf(&ctx.out,out)){+rv=-1;+gotodone;+}done:strbuf_release(&ctx.data);strbuf_release(&ctx.instructions);
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:46
Check that the declared output size for each window is correct, and
reserve that amount of space in the output buffer in advance.
Signed-off-by: Jonathan Nieder <redacted>
---
vcs-svn/svndiff.c | 5 +++++
1 files changed, 5 insertions(+), 0 deletions(-)
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:46
The main point is to constrain the format of deltas more,
so corruption and other breakage can be more easily detected.
Requiring deltas not to provide unconsumed data also opens
the possibility of ignoring the declared amount of new data
and simply streaming the data as needed to fulfill
copyfrom_data requests.
Signed-off-by: Jonathan Nieder <redacted>
---
t/t9011-svn-da.sh | 12 +++---------
vcs-svn/svndiff.c | 2 ++
2 files changed, 5 insertions(+), 9 deletions(-)
@@ -188,6 +188,8 @@ static int apply_window_in_core(struct window *ctx)while(insn!=ctx->instructions.buf+ctx->instructions.len)if(step(ctx,&insn,&data_pos))return-1;+if(data_pos!=ctx->data.len)+returnerror("Invalid delta: does not copy all new data");return0;}
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:46
The copyfrom_target instruction copies appends data that is already
present in the current output view to the end of output. (The offset
argument is relative to the beginning of output produced by the
current window.)
The region copied is allowed to run past the end of the existing
output. To support that case, copy one character at a time
rather than using memcpy() or memmove(). This allows copyfrom_target
to be used once to repeat a string many times. For example:
COPYFROM_DATA 2
COPYFROM_OUTPUT 10, 0
DATA "ab"
would produce the output "ababababababababababab".
Signed-off-by: Jonathan Nieder <redacted>
---
t/t9011-svn-da.sh | 42 ++++++++++++++++++++++++++++++++++++++++++
vcs-svn/svndiff.c | 30 ++++++++++++++++++++++++++++--
2 files changed, 70 insertions(+), 2 deletions(-)
@@ -159,4 +159,46 @@ test_expect_success 'catch attempt to copy missing data' 'test_must_failtest-svn-fe-dpreimagecopy.incomplete$len'+test_expect_success'copyfrom target to repeat data''+printffoofoo>expect&&+printf"SVNQ%b%b%s""QQ\006\004\003""\0203\0100\003Q""foo"|+q_to_nul>copytarget.repeat&&+len=$(wc-c<copytarget.repeat)&&+test-svn-fe-dpreimagecopytarget.repeat$len>actual&&+test_cmpexpectactual+'++test_expect_success'copyfrom target out of order''+printffoooof>expect&&+printf"SVNQ%b%b%s"\+"QQ\006\007\003""\0203\0101\002\0101\001\0101Q""foo"|+q_to_nul>copytarget.reverse&&+len=$(wc-c<copytarget.reverse)&&+test-svn-fe-dpreimagecopytarget.reverse$len>actual&&+test_cmpexpectactual+'++test_expect_success'catch copyfrom future''+printf"SVNQ%b%b%s""QQ\004\004\003""\0202\0101\002\0201""XYZ"|+q_to_nul>copytarget.infuture&&+len=$(wc-c<copytarget.infuture)&&+test_must_failtest-svn-fe-dpreimagecopytarget.infuture$len+'++test_expect_success'copy to sustain''+printfXYXYXYXYXYXZ>expect&&+printf"SVNQ%b%b%s""QQ\014\004\003""\0202\0111Q\0201""XYZ"|+q_to_nul>copytarget.sustain&&+len=$(wc-c<copytarget.sustain)&&+test-svn-fe-dpreimagecopytarget.sustain$len>actual&&+test_cmpexpectactual+'++test_expect_success'catch copy that overflows''+printf"SVNQ%b%b%s""QQ\003\003\001""\0201\0177Q"X|+q_to_nul>copytarget.overflow&&+len=$(wc-c<copytarget.overflow)&&+test_must_failtest-svn-fe-dpreimagecopytarget.overflow$len+'+ test_done
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:46
Some particularly strange deltas of unknown origin were found to
request copies beyond the end of the preimage. But svn 1.6 never
produces anything like that.
Although Subversion accepts these perverse deltas as input, let's
error out if some future version of subversion starts to actually
produce them.
Without this change, the diff applier would have to separately
keep track of the number of bytes supposedly and actually written out.
Helped-by: Ramkumar Ramachandra [off-list ref]
Helped-by: David Barr [off-list ref]
Signed-off-by: Jonathan Nieder <redacted>
---
t/t9011-svn-da.sh | 11 ++++-------
vcs-svn/sliding_window.c | 10 ++++++----
2 files changed, 10 insertions(+), 11 deletions(-)
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:46
The copyfrom_source instruction appends data from the preimage
buffer to the end of output. Its arguments are a length and an
offset relative to the beginning of the source view.
Helped-by: Ramkumar Ramachandra [off-list ref]
Helped-by: David Barr [off-list ref]
Signed-off-by: Jonathan Nieder <redacted>
---
That's the end of the series. Thanks for reading. Hopefully this
round did not introduce too many bugs but if it did, I'd be glad to
hear about them.
Good night,
Jonathan
t/t9011-svn-da.sh | 35 +++++++++++++++++++++++++++++++++++
vcs-svn/svndiff.c | 27 +++++++++++++++++++++++----
2 files changed, 58 insertions(+), 4 deletions(-)
@@ -198,4 +198,39 @@ test_expect_success 'catch copy that overflows' 'test_must_failtest-svn-fe-dpreimagecopytarget.overflow$len'+test_expect_success'copyfrom source''+printffoo>expect&&+printf"SVNQ%b%b""Q\003\003\002Q""\003Q"|q_to_nul>copysource.all&&+test-svn-fe-dpreimagecopysource.all11>actual&&+test_cmpexpectactual+'++test_expect_success'copy backwards''+printfoof>expect&&+printf"SVNQ%b%b""Q\003\003\006Q""\001\002\001\001\001Q"|+q_to_nul>copysource.rev&&+test-svn-fe-dpreimagecopysource.rev15>actual&&+test_cmpexpectactual+'++test_expect_success'offsets are relative to window''+printffo>expect&&+printf"SVNQ%b%b%b%b""Q\003\001\002Q""\001Q"\+"\002\001\001\002Q""\001Q"|+q_to_nul>copysource.two&&+test-svn-fe-dpreimagecopysource.two18>actual&&+test_cmpexpectactual+'++test_expect_success'example from notes/svndiff''+printfaaaaccccdddddddd>expect&&+printfaaaabbbbcccc>source&&+printf"SVNQ%b%b%s""Q\014\020\007\001"\+"\004Q\004\010\0201\0107\010"d|+q_to_nul>delta.example&&+len=$(wc-c<delta.example)&&+test-svn-fe-dsourcedelta.example$len>actual&&+test_cmpexpectactual+'+ test_done
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:46
The copyfrom_source instruction appends data from the preimage
buffer to the end of output. Its arguments are a length and an
offset relative to the beginning of the source view.
Helped-by: Ramkumar Ramachandra [off-list ref]
Helped-by: David Barr [off-list ref]
Signed-off-by: Jonathan Nieder <redacted>
---
[resending with cc: samv]
That's the end of the series. Thanks for reading. Hopefully this
round did not introduce too many bugs but if it did, I'd be glad to
hear about them.
Good night,
Jonathan
t/t9011-svn-da.sh | 35 +++++++++++++++++++++++++++++++++++
vcs-svn/svndiff.c | 27 +++++++++++++++++++++++----
2 files changed, 58 insertions(+), 4 deletions(-)
@@ -198,4 +198,39 @@ test_expect_success 'catch copy that overflows' 'test_must_failtest-svn-fe-dpreimagecopytarget.overflow$len'+test_expect_success'copyfrom source''+printffoo>expect&&+printf"SVNQ%b%b""Q\003\003\002Q""\003Q"|q_to_nul>copysource.all&&+test-svn-fe-dpreimagecopysource.all11>actual&&+test_cmpexpectactual+'++test_expect_success'copy backwards''+printfoof>expect&&+printf"SVNQ%b%b""Q\003\003\006Q""\001\002\001\001\001Q"|+q_to_nul>copysource.rev&&+test-svn-fe-dpreimagecopysource.rev15>actual&&+test_cmpexpectactual+'++test_expect_success'offsets are relative to window''+printffo>expect&&+printf"SVNQ%b%b%b%b""Q\003\001\002Q""\001Q"\+"\002\001\001\002Q""\001Q"|+q_to_nul>copysource.two&&+test-svn-fe-dpreimagecopysource.two18>actual&&+test_cmpexpectactual+'++test_expect_success'example from notes/svndiff''+printfaaaaccccdddddddd>expect&&+printfaaaabbbbcccc>source&&+printf"SVNQ%b%b%s""Q\014\020\007\001"\+"\004Q\004\010\0201\0107\010"d|+q_to_nul>delta.example&&+len=$(wc-c<delta.example)&&+test-svn-fe-dsourcedelta.example$len>actual&&+test_cmpexpectactual+'+ test_done
From: Sam Vilain <hidden> Date: 2016-06-15 22:49:47
On Wed, 2010-10-13 at 04:30 -0500, Jonathan Nieder wrote:
It occurs to me that Sam Vilain may well have something valuable to
say about this series, having implemented something similar[1].
Sam, this series adds an svndiff0 parser for git to use in parsing
v3 dumps (which are way easier to produce with remote access to an
svn repository than v2 dumps). The beginning of the series is at [2],
though that cover letter is out of date: now, modulo any new bugs
I've introduced with this reroll, it is known to successfully apply
all the deltas involved in a complete dump of the ASF repo.
All I did was make the module lazy - version 0.02 was by 唐鳳, any
detailed knowledge I had of the binary format fell out of my head pretty
quickly :-)
Sam
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:47
Sam Vilain wrote:
All I did was make the module lazy - version 0.02 was by 唐鳳, any
detailed knowledge I had of the binary format fell out of my head pretty
quickly :-)
Ah, my bad. 唐鳳, as part of an attempt to natively support fetching
from and pushing to svn repositories, some contributors to the git
project are working on an svndiff0 applier. If you're interested in
reliving old memories, please feel free to look it over (especially
the test cases). Thoughts, simplifications, bug reports, improvements
welcome.
http://thread.gmane.org/gmane.comp.version-control.git/151086/focus=158913
Anyone wanting to try it can check out
git://repo.or.cz/git/jrn.git svn-da
and use the test-svn-fe command:
make test-svn-fe
./test-svn-fe -d <preimage> <delta> <delta length>
or the tests:
make
cd t && sh t9011-svn-da.sh -v -i
The preimage or delta argument can be /dev/stdin for use in a pipeline.
Thanks for your work on svk and pugs!
Jonathan