From: David Barr <hidden> Date: 2016-06-15 22:50:48
This eliminates one more dependency on string_pool.
Signed-off-by: David Barr <redacted>
Signed-off-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
---
vcs-svn/svndump.c | 50 ++++++++++++++++++++++++++++++++------------------
1 files changed, 32 insertions(+), 18 deletions(-)
@@ -236,8 +236,7 @@ static void handle_node(void)old_data=NULL;}elseif(node_ctx.action==NODEACT_CHANGE){uint32_tmode;-old_data=repo_read_path(node_ctx.dst.buf);-mode=repo_read_mode(node_ctx.dst.buf);+old_data=repo_read_path(node_ctx.dst.buf,&mode);if(mode==REPO_MODE_DIR&&type!=REPO_MODE_DIR)die("invalid dump: cannot modify a directory into a file");if(mode!=REPO_MODE_DIR&&type==REPO_MODE_DIR)
@@ -1,43 +0,0 @@-string_pool API-===============--The string_pool API provides facilities for replacing strings-with integer keys that can be more easily compared and stored.-The facilities are designed so that one could teach Git without-too much trouble to store the information needed for these keys to-remain valid over multiple executions.--Functions------------pool_intern::- Include a string in the string pool and get its key.- If that string is already in the pool, retrieves its- existing key.--pool_fetch::- Retrieve the string associated to a given key.--pool_tok_r::- Extract the key of the next token from a string.- Interface mimics strtok_r.--pool_print_seq::- Print a sequence of strings named by key to a file, using the- specified delimiter to separate them.-- If NULL (key ~0) appears in the sequence, the sequence ends- early.--pool_tok_seq::- Split a string into tokens, storing the keys of segments- into a caller-provided array.-- Unless sz is 0, the array will always be ~0-terminated.- If there is not enough room for all the tokens, the- array holds as many tokens as fit in the entries before- the terminating ~0. Return value is the index after the- last token, or sz if the tokens did not fit.--pool_reset::- Deallocate storage for the string pool.
From: David Barr <hidden> Date: 2016-06-15 22:50:48
This eliminates one more dependency on string_pool.
Signed-off-by: David Barr <redacted>
Signed-off-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
---
vcs-svn/svndump.c | 110 +++++++++++++++++++++++++++++------------------------
1 files changed, 60 insertions(+), 50 deletions(-)
@@ -1,9 +1,6 @@ Copyright (C) 2010 David Barr <david.barr@cordelta.com>. All rights reserved.-Copyright (C) 2008 Jason Evans <jasone@canonware.com>.-All rights reserved.- Copyright (C) 2005 Stefan Hegny, hydrografix Consulting GmbH, Frankfurt/Main, Germany and others, see http://svn2cc.sarovar.org
@@ -1,109 +0,0 @@-Motivation-==========--Treaps provide a memory-efficient binary search tree structure.-Insertion/deletion/search are about as about as fast in the average-case as red-black trees and the chances of worst-case behavior are-vanishingly small, thanks to (pseudo-)randomness. The bad worst-case-behavior is a small price to pay, given that treaps are much simpler-to implement.--API-===--The trp API generates a data structure and functions to handle a-large growing set of objects stored in a pool.--The caller:--. Specifies parameters for the generated functions with the- trp_gen(static, foo_, ...) macro.--. Allocates a `struct trp_root` variable and sets it to {~0}.--. Adds new nodes to the set using `foo_insert`. Any pointers- to existing nodes cannot be relied upon any more, so the caller- might retrieve them anew with `foo_pointer`.--. Can find a specific item in the set using `foo_search`.--. Can iterate over items in the set using `foo_first` and `foo_next`.--. Can remove an item from the set using `foo_remove`.--Example:-------struct ex_node {- const char *s;- struct trp_node ex_link;-};-static struct trp_root ex_base = {~0};-obj_pool_gen(ex, struct ex_node, 4096);-trp_gen(static, ex_, struct ex_node, ex_link, ex, strcmp)-struct ex_node *item;--item = ex_pointer(ex_alloc(1));-item->s = "hello";-ex_insert(&ex_base, item);-item = ex_pointer(ex_alloc(1));-item->s = "goodbye";-ex_insert(&ex_base, item);-for (item = ex_first(&ex_base); item; item = ex_next(&ex_base, item))- printf("%s\n", item->s);-------Functions------------trp_gen(attr, foo_, node_type, link_field, pool, cmp)::-- Generate a type-specific treap implementation.-+-. The storage class for generated functions will be 'attr' (e.g., `static`).-. Generated function names are prefixed with 'foo_' (e.g., `treap_`).-. Treap nodes will be of type 'node_type' (e.g., `struct treap_node`).- This type must be a struct with at least one `struct trp_node` field- to point to its children.-. The field used to access child nodes will be 'link_field'.-. All treap nodes must lie in the 'pool' object pool.-. Treap nodes must be totally ordered by the 'cmp' relation, with the- following prototype:-+-int (*cmp)(node_type \*a, node_type \*b)-+-and returning a value less than, equal to, or greater than zero-according to the result of comparison.--node_type {asterisk}foo_insert(struct trp_root *treap, node_type \*node)::-- Insert node into treap. If inserted multiple times,- a node will appear in the treap multiple times.-+-The return value is the address of the node within the treap,-which might differ from `node` if `pool_alloc` had to call-`realloc` to expand the pool.--void foo_remove(struct trp_root *treap, node_type \*node)::-- Remove node from treap. Caller must ensure node is- present in treap before using this function.--node_type *foo_search(struct trp_root \*treap, node_type \*key)::-- Search for a node that matches key. If no match is found,- result is NULL.--node_type *foo_nsearch(struct trp_root \*treap, node_type \*key)::-- Like `foo_search`, but if if the key is missing return what- would be key's successor, were key in treap (NULL if no- successor).--node_type *foo_first(struct trp_root \*treap)::-- Find the first item from the treap, in sorted order.--node_type *foo_next(struct trp_root \*treap, node_type \*node)::-- Find the next item.
From: David Barr <hidden> Date: 2016-06-15 22:50:48
This is a small optimisation (4% reduction in user time) but is the largest
artifact within the parsing portion of svndump.c
Signed-off-by: David Barr <redacted>
Signed-off-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
---
vcs-svn/svndump.c | 4 +++-
1 files changed, 3 insertions(+), 1 deletions(-)
From: David Barr <hidden> Date: 2016-06-15 22:50:48
Now that there is no internal representation of the repo,
it is not necessary to tokenise paths.
Use strbuf instead and bypass string_pool.
Signed-off-by: David Barr <redacted>
Signed-off-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
---
vcs-svn/fast_export.c | 47 ++++++++++++++++++------------------
vcs-svn/fast_export.h | 9 +++----
vcs-svn/repo_tree.c | 20 +++++++-------
vcs-svn/repo_tree.h | 13 ++++------
vcs-svn/svndump.c | 63 +++++++++++++++++++++----------------------------
5 files changed, 70 insertions(+), 82 deletions(-)
@@ -8,18 +8,17 @@ void fast_export_init(int fd);voidfast_export_deinit(void);voidfast_export_reset(void);-voidfast_export_delete(uint32_tdepth,constuint32_t*path);-voidfast_export_modify(uint32_tdepth,constuint32_t*path,-uint32_tmode,constchar*dataref);+voidfast_export_delete(constchar*path);+voidfast_export_modify(constchar*path,uint32_tmode,constchar*dataref);voidfast_export_begin_commit(uint32_trevision,uint32_tauthor,char*log,uint32_tuuid,uint32_turl,unsignedlongtimestamp);voidfast_export_end_commit(uint32_trevision);voidfast_export_data(uint32_tmode,uint32_tlen,structline_buffer*input);/* If there is no such file at that rev, returns -1, errno == ENOENT. */-intfast_export_ls_rev(uint32_trev,uint32_tdepth,constuint32_t*path,+intfast_export_ls_rev(uint32_trev,constchar*path,uint32_t*mode_out,structstrbuf*dataref_out);-intfast_export_ls(uint32_tdepth,constuint32_t*path,+intfast_export_ls(constchar*path,uint32_t*mode_out,structstrbuf*dataref_out);#endif
@@ -223,14 +213,14 @@ static void handle_node(void)if(have_text||have_props||node_ctx.srcRev)die("invalid dump: deletion node has ""copyfrom info, text, or properties");-returnrepo_delete(node_ctx.dst);+returnrepo_delete(node_ctx.dst.buf);}if(node_ctx.action==NODEACT_REPLACE){-repo_delete(node_ctx.dst);+repo_delete(node_ctx.dst.buf);node_ctx.action=NODEACT_ADD;}if(node_ctx.srcRev){-repo_copy(node_ctx.srcRev,node_ctx.src,node_ctx.dst);+repo_copy(node_ctx.srcRev,node_ctx.src.buf,node_ctx.dst.buf);if(node_ctx.action==NODEACT_ADD)node_ctx.action=NODEACT_CHANGE;}
@@ -240,14 +230,14 @@ static void handle_node(void)/**Findoldcontent(old_data)anddecideonthenewmode.*/-if(node_ctx.action==NODEACT_CHANGE&&!~*node_ctx.dst){+if(node_ctx.action==NODEACT_CHANGE&&!*node_ctx.dst.buf){if(type!=REPO_MODE_DIR)die("invalid dump: root of tree is not a regular file");old_data=NULL;}elseif(node_ctx.action==NODEACT_CHANGE){uint32_tmode;-old_data=repo_read_path(node_ctx.dst);-mode=repo_read_mode(node_ctx.dst);+old_data=repo_read_path(node_ctx.dst.buf);+mode=repo_read_mode(node_ctx.dst.buf);if(mode==REPO_MODE_DIR&&type!=REPO_MODE_DIR)die("invalid dump: cannot modify a directory into a file");if(mode!=REPO_MODE_DIR&&type==REPO_MODE_DIR)
@@ -284,12 +274,10 @@ static void handle_node(void)/* For the fast_export_* functions, NULL means empty. */old_data=NULL;if(!have_text){-fast_export_modify(REPO_MAX_PATH_DEPTH,node_ctx.dst,-node_ctx.type,old_data);+fast_export_modify(node_ctx.dst.buf,node_ctx.type,old_data);return;}-fast_export_modify(REPO_MAX_PATH_DEPTH,node_ctx.dst,-node_ctx.type,"inline");+fast_export_modify(node_ctx.dst.buf,node_ctx.type,"inline");fast_export_data(node_ctx.type,node_ctx.textLength,&input);}
From: David Barr <hidden> Date: 2016-06-15 22:50:48
This series is largely the work of Jonathan Nieder.
I have painstakingly ported it from the old development branch to the
latest series to hit the list. As previously, I have tested against
the ASF subversion repository to increase confidence in the series.
Hopefully, this brings us a little closer to having full support for
version 3 of the subversion dump format in master.
Makefile | 5 +-
contrib/svn-fe/svn-fe.txt | 5 +-
t/t9010-svn-fe.sh | 108 ++++++++++++++++-
t/t9011-svn-da.sh | 250 ++++++++++++++++++++++++++++++++++++
test-svn-fe.c | 42 +++++--
vcs-svn/LICENSE | 2 +
vcs-svn/fast_export.c | 122 +++++++++++++++++-
vcs-svn/fast_export.h | 3 +
vcs-svn/line_buffer.c | 37 +++---
vcs-svn/line_buffer.h | 7 +-
vcs-svn/line_buffer.txt | 3 +-
vcs-svn/sliding_window.c | 74 +++++++++++
vcs-svn/sliding_window.h | 17 +++
vcs-svn/svndiff.c | 308 +++++++++++++++++++++++++++++++++++++++++++++
vcs-svn/svndiff.h | 10 ++
vcs-svn/svndump.c | 35 ++++-
16 files changed, 980 insertions(+), 48 deletions(-)
From: David Barr <hidden> Date: 2016-06-15 22:50:48
From: Jonathan Nieder <redacted>
Currently there is no way to detect when input ended if it ended
early during buffer_skip_bytes. Tell the calling program how many
bytes were actually skipped for easier debugging.
Existing callers will still ignore early EOF.
Signed-off-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
---
vcs-svn/line_buffer.c | 13 +++++++------
vcs-svn/line_buffer.h | 2 +-
vcs-svn/line_buffer.txt | 3 ++-
3 files changed, 10 insertions(+), 8 deletions(-)
@@ -76,7 +76,8 @@ Functions `buffer_skip_bytes`:: Discards `len` bytes from the input stream (stopping early- if necessary because of an error or eof).+ if necessary because of an error or eof). Return value is+ the number of bytes successfully read. `buffer_reset`:: Deallocates non-static buffers.
From: David Barr <hidden> Date: 2016-06-15 22:50:48
From: Jonathan Nieder <redacted>
Each section of a Subversion-format delta only requires examining (and
keeping in random-access memory) a small portion of the preimage. At
any moment, this portion is starts at a certain file offset and has a
well-defined length, and as the delta is applied, the portion moves
from the beginning to the end of the preimage. Add a move_window
function to keep track of this view into the preimage.
You can use it like this:
buffer_init(f, NULL);
struct sliding_view window = SLIDING_VIEW_INIT(f);
move_window(&window, 3, 7); /* (1) */
move_window(&window, 5, 5); /* (2) */
move_window(&window, 12, 2); /* (3) */
strbuf_release(&window.buf);
buffer_deinit(f);
The data structure is called sliding_view instead of _window to
prevent confusion with svndiff0 Windows.
In this example, (1) reads 10 bytes and discards the first 3;
(2) discards the first 2, which are not needed any more; and (3) skips
2 bytes and reads 2 new bytes to work with.
When move_window returns, the file position indicator is at position
window->off + window->width and the data from positions window->off to
the current file position are stored in window->buf.
This function performs only sequential access from the input file and
never seeks, so it can be safely used on pipes and sockets.
On end-of-file, move_window silently reads less than the caller
requested. On other errors, it prints a message and returns -1.
Helped-by: David Barr [off-list ref]
Signed-off-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
---
Makefile | 5 ++-
vcs-svn/LICENSE | 2 +
vcs-svn/sliding_window.c | 74 ++++++++++++++++++++++++++++++++++++++++++++++
vcs-svn/sliding_window.h | 17 ++++++++++
4 files changed, 96 insertions(+), 2 deletions(-)
create mode 100644 vcs-svn/sliding_window.c
create mode 100644 vcs-svn/sliding_window.h
@@ -1,6 +1,8 @@ Copyright (C) 2010 David Barr <david.barr@cordelta.com>. All rights reserved.+Copyright (C) 2010 Jonathan Nieder <jrnieder@gmail.com>.+ Copyright (C) 2005 Stefan Hegny, hydrografix Consulting GmbH, Frankfurt/Main, Germany and others, see http://svn2cc.sarovar.org
From: David Barr <hidden> Date: 2016-06-15 22:50:48
From: Jonathan Nieder <redacted>
Currently buffer_copy_bytes does not report to its caller whether
it encountered an early end of file.
Add a return value representing the number of bytes read (but not
the number of bytes copied). This way all three unusual conditions
can be distinguished: input error with buffer_ferror, output error
with ferror(outfile), early end of input by checking the return
value.
Signed-off-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
---
vcs-svn/line_buffer.c | 18 +++++++++---------
vcs-svn/line_buffer.h | 3 ++-
2 files changed, 11 insertions(+), 10 deletions(-)
From: David Barr <hidden> Date: 2016-06-15 22:50:48
From: Jonathan Nieder <redacted>
Catch input errors and exit early enough to print a reasonable
diagnosis based on errno.
Signed-off-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
---
vcs-svn/fast_export.c | 13 +++++++++++--
vcs-svn/svndump.c | 22 +++++++++++++++++++---
2 files changed, 30 insertions(+), 5 deletions(-)
From: David Barr <hidden> Date: 2016-06-15 22:50:48
From: Jonathan Nieder <redacted>
Move from uint32_t to off_t as the fundamental unit of length used by
the line_buffer library. Performance would get worse if anything but
I think it's worth it for support of deltas that need to skip large
pieces (> 4 GiB).
Exception: buffer_read_string still takes a uint32_t, since it keeps
its result in an in-core obj_pool.
Callers still have to be updated to take advantage of this.
Signed-off-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
---
vcs-svn/line_buffer.c | 8 ++++----
vcs-svn/line_buffer.h | 4 ++--
2 files changed, 6 insertions(+), 6 deletions(-)
From: David Barr <hidden> Date: 2016-06-15 22:50:48
From: Jonathan Nieder <redacted>
Buffer the instruction section upon encountering it for later
interpretation.
An alternative design would involve parsing the instructions
at this point and buffering them in some processed form. Using
the unprocessed form is simpler.
Signed-off-by: Jonathan Nieder <redacted>
Acked-by: Ramkumar Ramachandra <redacted>
Signed-off-by: David Barr <redacted>
---
t/t9011-svn-da.sh | 5 +++++
vcs-svn/svndiff.c | 6 +++++-
2 files changed, 10 insertions(+), 1 deletions(-)
@@ -126,6 +128,8 @@ static int apply_one_window(struct line_buffer *delta, off_t *delta_len)read_length(delta,&instructions_len,delta_len)||read_length(delta,&data_len,delta_len))gotoerror_out;+if(read_chunk(delta,delta_len,&ctx.instructions,instructions_len))+gotoerror_out;if(instructions_len){error("What do you think I am? A delta applier?");gotoerror_out;
From: David Barr <hidden> Date: 2016-06-15 22:50:48
From: Jonathan Nieder <redacted>
A delta in the subversion delta (svndiff0) format consists of the
magic bytes SVN\0 followed by a sequence of windows of a certain
well specified format (starting with five integers).
Add an svndiff0_apply function and test-svn-fe -d commandline tool to
parse such a delta in the special case of not including any windows.
Later patches will add features to turn this into a fully functional
delta applier for use by svn-fe in parsing the streams produced by
"svnrdump dump" and "svnadmin dump --deltas".
The content of symlinks starts with the word "link " in Subversion's
worldview, so we will need to prepend that text for the sake of
delta application. Initialization of the input state of the
delta preimage is left to the calling program, which gives callers
a chance to seed the sliding window with text of their choice.
Improved-by: Ramkumar Ramachandra [off-list ref]
Improved-by: David Barr [off-list ref]
Signed-off-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
---
Makefile | 4 +-
t/t9011-svn-da.sh | 37 ++++++++++++++++++++++++++++++++++
test-svn-fe.c | 42 ++++++++++++++++++++++++++++++++-------
vcs-svn/line_buffer.c | 6 ++--
vcs-svn/line_buffer.h | 2 +-
vcs-svn/svndiff.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++++
vcs-svn/svndiff.h | 10 +++++++++
7 files changed, 139 insertions(+), 14 deletions(-)
create mode 100755 t/t9011-svn-da.sh
create mode 100644 vcs-svn/svndiff.c
create mode 100644 vcs-svn/svndiff.h
@@ -4,15 +4,41 @@#include"git-compat-util.h"#include"vcs-svn/svndump.h"+#include"vcs-svn/svndiff.h"+#include"vcs-svn/sliding_window.h"+#include"vcs-svn/line_buffer.h"intmain(intargc,char*argv[]){-if(argc!=2)-usage("test-svn-fe <file>");-if(svndump_init(argv[1]))-return1;-svndump_read(NULL);-svndump_deinit();-svndump_reset();-return0;+staticconstchartest_svnfe_usage[]=+"test-svn-fe (<dumpfile> | [-d] <preimage> <delta> <len>)";+if(argc==2){+if(svndump_init(argv[1]))+return1;+svndump_read(NULL);+svndump_deinit();+svndump_reset();+return0;+}+if(argc==5&&!strcmp(argv[1],"-d")){+structline_bufferpreimage=LINE_BUFFER_INIT;+structline_bufferdelta=LINE_BUFFER_INIT;+structsliding_viewpreimage_view=SLIDING_VIEW_INIT(&preimage);+if(buffer_init(&preimage,argv[2]))+die_errno("cannot open preimage");+if(buffer_init(&delta,argv[3]))+die_errno("cannot open delta");+if(svndiff0_apply(&delta,(off_t)strtoull(argv[4],NULL,0),+&preimage_view,stdout))+return1;+if(buffer_deinit(&preimage))+die_errno("cannot close preimage");+if(buffer_deinit(&delta))+die_errno("cannot close delta");+buffer_reset(&preimage);+strbuf_release(&preimage_view.buf);+buffer_reset(&delta);+return0;+}+usage(test_svnfe_usage);}
@@ -0,0 +1,52 @@+/*+*Licensedunderatwo-clauseBSD-stylelicense.+*SeeLICENSEfordetails.+*/++#include"git-compat-util.h"+#include"line_buffer.h"+#include"svndiff.h"++/*+*svndiff0applier+*+*Seehttp://svn.apache.org/repos/asf/subversion/trunk/notes/svndiff.+*+*svndiff0::='SVN\0'window*+*/++staticinterror_short_read(structline_buffer*input)+{+if(buffer_ferror(input))+returnerror("error reading delta: %s",strerror(errno));+returnerror("invalid delta: unexpected end of file");+}++staticintread_magic(structline_buffer*in,off_t*len)+{+staticconstcharmagic[]={'S','V','N','\0'};+structstrbufsb=STRBUF_INIT;++if(*len<sizeof(magic)||+buffer_read_binary(in,&sb,sizeof(magic))!=sizeof(magic))+returnerror_short_read(in);++if(memcmp(sb.buf,magic,sizeof(magic)))+returnerror("invalid delta: unrecognized file type");++*len-=sizeof(magic);+strbuf_release(&sb);+return0;+}++intsvndiff0_apply(structline_buffer*delta,off_tdelta_len,+structsliding_view*preimage,FILE*postimage)+{+assert(delta&&preimage&&postimage);++if(read_magic(delta,&delta_len))+return-1;+if(delta_len)+returnerror("What do you think I am? A delta applier?");+return0;+}
From: David Barr <hidden> Date: 2016-06-15 22:50:48
From: Jonathan Nieder <redacted>
Each window of an svndiff0-format delta includes a section for new
data that will be copied into the preimage (in the order it appears in
the window, possibly interspersed with other data).
Read this data when encountering it. It is not actually necessary to
do so --- it would be just as easy to copy straight from the delta
to output when interpreting the relevant instructions --- but this
way, the code that interprets svndiff0 instructions can proceed more
quickly because it does not require any I/O.
Subversion's implementation rejects deltas that do not consume all
the auxiliary data that is available. Do not check that for now,
because it would make it impossible to test the function of this
patch until the instructions to consume data are implemented.
Do check for truncated data sections. Since Subversion's applier
rejects deltas that end before the new-data section is declared to
end, it should be safe for this applier to reject such deltas, too.
Improved-by: Ramkumar Ramachandra [off-list ref]
Improved-by: David Barr [off-list ref]
Signed-off-by: Jonathan Nieder <redacted>
Acked-by: Ramkumar Ramachandra <redacted>
Signed-off-by: David Barr <redacted>
---
t/t9011-svn-da.sh | 12 ++++++++++++
vcs-svn/svndiff.c | 46 +++++++++++++++++++++++++++++++++++-----------
2 files changed, 47 insertions(+), 11 deletions(-)
@@ -107,12 +125,18 @@ static int apply_one_window(struct line_buffer *delta, off_t *delta_len)if(read_length(delta,&out_len,delta_len)||read_length(delta,&instructions_len,delta_len)||read_length(delta,&data_len,delta_len))-return-1;-if(instructions_len)-returnerror("What do you think I am? A delta applier?");-if(data_len)-returnerror("No support for inline data yet");+gotoerror_out;+if(instructions_len){+error("What do you think I am? A delta applier?");+gotoerror_out;+}+if(read_chunk(delta,delta_len,&ctx.data,data_len))+gotoerror_out;+window_release(&ctx);return0;+error_out:+window_release(&ctx);+return-1;}intsvndiff0_apply(structline_buffer*delta,off_tdelta_len,
From: David Barr <hidden> Date: 2016-06-15 22:50:49
From: Jonathan Nieder <redacted>
The source view offset heading each svndiff0 window represents a
number of bytes past the beginning of the preimage. Together with the
source view length, it dictates to the delta applier what portion of
the preimage instructions will refer to. Read that portion right away
using the sliding window code.
Maybe some day we will use mmap to read data more lazily.
Subversion's implementation tolerates source view offsets pointing
past the end of the preimage file but we do not, for simplicity.
This does not teach the delta applier to read instructions or copy
data from the source view. Deltas that could produce nonempty output
will still be rejected.
Improved-by: Ramkumar Ramachandra [off-list ref]
Improved-by: David Barr [off-list ref]
Signed-off-by: Jonathan Nieder <redacted>
Acked-by: Ramkumar Ramachandra <redacted>
Signed-off-by: David Barr <redacted>
---
t/t9011-svn-da.sh | 35 +++++++++++++++++++++++++++++++++++
vcs-svn/svndiff.c | 6 ++++--
2 files changed, 39 insertions(+), 2 deletions(-)
@@ -122,11 +123,12 @@ int svndiff0_apply(struct line_buffer *delta, off_t delta_len,if(read_magic(delta,&delta_len))return-1;while(delta_len){/* For each window: */-off_tpre_off;-size_tpre_len;+off_tpre_off=pre_off;+size_tpre_len=pre_len;if(read_offset(delta,&pre_off,&delta_len)||read_length(delta,&pre_len,&delta_len)||+move_window(preimage,pre_off,pre_len)||apply_one_window(delta,&delta_len))return-1;}
From: David Barr <hidden> Date: 2016-06-15 22:50:49
From: Jonathan Nieder <redacted>
Combine some ifs. No functional change intended.
Missed this in "vcs-svn: let deltas use data from preimage"
(2010-10-13).
Signed-off-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
---
vcs-svn/svndiff.c | 8 +++-----
1 files changed, 3 insertions(+), 5 deletions(-)
From: David Barr <hidden> Date: 2016-06-15 22:50:49
From: Jonathan Nieder <redacted>
The copyfrom_source instruction appends data from the preimage buffer
to the end of output. Its arguments are a length and an offset
relative to the beginning of the source view.
With this change, the delta applier is able to reproduce all 5,636,613
blobs in the early history of the ASF repository. Tested with
mkfifo backflow
svn-fe <svn-asf-public-r0:940166 3<backflow |
git fast-import --cat-blob-fd=3 3>backflow
with svn-asf-public-r0:940166 produced by whatever version of
Subversion the dumps in /dump/ on svn.apache.org use (presumably
1.6.something).
Improved-by: Ramkumar Ramachandra [off-list ref]
Improved-by: David Barr [off-list ref]
Signed-off-by: Jonathan Nieder <redacted>
Acked-by: Ramkumar Ramachandra <redacted>
Signed-off-by: David Barr <redacted>
---
t/t9011-svn-da.sh | 35 +++++++++++++++++++++++++++++++++++
vcs-svn/svndiff.c | 28 +++++++++++++++++++++++-----
2 files changed, 58 insertions(+), 5 deletions(-)
@@ -212,4 +212,39 @@ test_expect_success 'catch copy that overflows' 'test_must_failtest-svn-fe-dpreimagecopytarget.overflow$len'+test_expect_success'copyfrom source''+printffoo>expect&&+printf"SVNQ%b%b""Q\003\003\002Q""\003Q"|q_to_nul>copysource.all&&+test-svn-fe-dpreimagecopysource.all11>actual&&+test_cmpexpectactual+'++test_expect_success'copy backwards''+printfoof>expect&&+printf"SVNQ%b%b""Q\003\003\006Q""\001\002\001\001\001Q"|+q_to_nul>copysource.rev&&+test-svn-fe-dpreimagecopysource.rev15>actual&&+test_cmpexpectactual+'++test_expect_success'offsets are relative to window''+printffo>expect&&+printf"SVNQ%b%b%b%b""Q\003\001\002Q""\001Q"\+"\002\001\001\002Q""\001Q"|+q_to_nul>copysource.two&&+test-svn-fe-dpreimagecopysource.two18>actual&&+test_cmpexpectactual+'++test_expect_success'example from notes/svndiff''+printfaaaaccccdddddddd>expect&&+printfaaaabbbbcccc>source&&+printf"SVNQ%b%b%s""Q\014\020\007\001"\+"\004Q\004\010\0201\0107\010"d|+q_to_nul>delta.example&&+len=$(wc-c<delta.example)&&+test-svn-fe-dsourcedelta.example$len>actual&&+test_cmpexpectactual+'+ test_done
From: David Barr <hidden> Date: 2016-06-15 22:50:49
From: Jonathan Nieder <redacted>
By constraining the format of deltas, we can more easily detect
corruption and other breakage.
Requiring deltas not to provide unconsumed data also opens the
possibility of ignoring the declared amount of new data and simply
streaming the data as needed to fulfill copyfrom_data requests.
Signed-off-by: Jonathan Nieder <redacted>
Acked-by: Ramkumar Ramachandra <redacted>
Signed-off-by: David Barr <redacted>
---
t/t9011-svn-da.sh | 5 ++---
vcs-svn/svndiff.c | 2 ++
2 files changed, 4 insertions(+), 3 deletions(-)
@@ -208,6 +208,8 @@ static int apply_window_in_core(struct window *ctx))if(execute_one_instruction(ctx,&instructions,&data_pos))return-1;+if(data_pos!=ctx->data.len)+returnerror("invalid delta: does not copy all inline data");return0;}
From: David Barr <hidden> Date: 2016-06-15 22:50:49
From: Jonathan Nieder <redacted>
The copyfrom_data instruction copies a few bytes verbatim from the
auxiliary data section of a window to the postimage.
[jn: with memory leak fix from David]
Improved-by: David Barr [off-list ref]
Signed-off-by: Jonathan Nieder <redacted>
Acked-by: Ramkumar Ramachandra <redacted>
Signed-off-by: David Barr <redacted>
---
t/t9011-svn-da.sh | 31 ++++++++++++++
vcs-svn/svndiff.c | 114 ++++++++++++++++++++++++++++++++++++++++++++++++++---
2 files changed, 139 insertions(+), 6 deletions(-)
@@ -130,11 +227,16 @@ static int apply_one_window(struct line_buffer *delta, off_t *delta_len)gotoerror_out;if(read_chunk(delta,delta_len,&ctx.instructions,instructions_len))gotoerror_out;-if(instructions_len){-error("What do you think I am? A delta applier?");+if(read_chunk(delta,delta_len,&ctx.data,data_len))+gotoerror_out;+strbuf_grow(&ctx.out,out_len);+if(apply_window_in_core(&ctx))+gotoerror_out;+if(ctx.out.len!=out_len){+error("invalid delta: incorrect postimage length");gotoerror_out;}-if(read_chunk(delta,delta_len,&ctx.data,data_len))+if(write_strbuf(&ctx.out,out))gotoerror_out;window_release(&ctx);return0;
@@ -157,7 +259,7 @@ int svndiff0_apply(struct line_buffer *delta, off_t delta_len,if(read_offset(delta,&pre_off,&delta_len)||read_length(delta,&pre_len,&delta_len)||move_window(preimage,pre_off,pre_len)||-apply_one_window(delta,&delta_len))+apply_one_window(delta,&delta_len,postimage))return-1;}return0;
From: David Barr <hidden> Date: 2016-06-15 22:50:49
From: Jonathan Nieder <redacted>
Each window in a subversion delta (svndiff0-format file) starts with a
window header, consisting of five integers with variable-length
representation:
source view offset
source view length
output length
instructions length
auxiliary data length
Parse it. The result is not usable for deltas with nonempty postimage
yet; in fact, this only adds support for deltas without any
instructions or auxiliary data. This is a good place to stop, though,
since that little support lets us add some simple passing tests
concerning error handling to the test suite.
Improved-by: Ramkumar Ramachandra [off-list ref]
Improved-by: David Barr [off-list ref]
Signed-off-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
---
t/t9011-svn-da.sh | 56 +++++++++++++++++++++++++++++++-
vcs-svn/svndiff.c | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++---
2 files changed, 142 insertions(+), 6 deletions(-)
@@ -28,17 +36,84 @@ static int read_magic(struct line_buffer *in, off_t *len)structstrbufsb=STRBUF_INIT;if(*len<sizeof(magic)||-buffer_read_binary(in,&sb,sizeof(magic))!=sizeof(magic))-returnerror_short_read(in);+buffer_read_binary(in,&sb,sizeof(magic))!=sizeof(magic)){+error_short_read(in);+strbuf_release(&sb);+return-1;+}-if(memcmp(sb.buf,magic,sizeof(magic)))+if(memcmp(sb.buf,magic,sizeof(magic))){+strbuf_release(&sb);returnerror("invalid delta: unrecognized file type");+}*len-=sizeof(magic);strbuf_release(&sb);return0;}+staticintread_int(structline_buffer*in,uintmax_t*result,off_t*len)+{+uintmax_trv=0;+off_tsz;+for(sz=*len;sz;sz--){+constintch=buffer_read_char(in);+if(ch==EOF)+break;++rv<<=VLI_BITS_PER_DIGIT;+rv+=(ch&VLI_DIGIT_MASK);+if(ch&VLI_CONTINUE)+continue;++*result=rv;+*len=sz-1;+return0;+}+returnerror_short_read(in);+}++staticintread_offset(structline_buffer*in,off_t*result,off_t*len)+{+uintmax_tval;+if(read_int(in,&val,len))+return-1;+if(val>maximum_signed_value_of_type(off_t))+returnerror("unrepresentable offset in delta: %"PRIuMAX"",val);+*result=val;+return0;+}++staticintread_length(structline_buffer*in,size_t*result,off_t*len)+{+uintmax_tval;+if(read_int(in,&val,len))+return-1;+if(val>SIZE_MAX)+returnerror("unrepresentable length in delta: %"PRIuMAX"",val);+*result=val;+return0;+}++staticintapply_one_window(structline_buffer*delta,off_t*delta_len)+{+size_tout_len;+size_tinstructions_len;+size_tdata_len;+assert(delta_len);++/* "source view" offset and length already handled; */+if(read_length(delta,&out_len,delta_len)||+read_length(delta,&instructions_len,delta_len)||+read_length(delta,&data_len,delta_len))+return-1;+if(instructions_len)+returnerror("What do you think I am? A delta applier?");+if(data_len)+returnerror("No support for inline data yet");+return0;+}+intsvndiff0_apply(structline_buffer*delta,off_tdelta_len,structsliding_view*preimage,FILE*postimage){
@@ -46,7 +121,14 @@ int svndiff0_apply(struct line_buffer *delta, off_t delta_len,if(read_magic(delta,&delta_len))return-1;-if(delta_len)-returnerror("What do you think I am? A delta applier?");+while(delta_len){/* For each window: */+off_tpre_off;+size_tpre_len;++if(read_offset(delta,&pre_off,&delta_len)||+read_length(delta,&pre_len,&delta_len)||+apply_one_window(delta,&delta_len))+return-1;+}return0;}
From: David Barr <hidden> Date: 2016-06-15 22:50:49
Handle input in Subversion's dumpfile format, version 3. This is the
format produced by "svnrdump dump" and "svnadmin dump --deltas", and
the main difference between v3 dumpfiles and the dumpfiles already
handled is that these can include nodes whose properties and text are
expressed relative to some other node.
To handle such nodes, we find which node the text and properties are
based on, handle its property changes, use the cat-blob command to
request the basis blob from the fast-import backend, use the
svndiff0_apply() helper to apply the text delta on the fly, writing
output to a temporary file, and then measure that postimage file's
length and write its content to the fast-import stream.
The temporary postimage file is shared between delta-using nodes to
avoid some file system overhead.
The svn-fe interface needs to be more complicated to accomodate the
backward flow of information from the fast-import backend to svn-fe.
The backflow fd is not needed when parsing streams without deltas,
though, so existing scripts using svn-fe on v2 dumps should
continue to work.
NEEDSWORK: generalize interface so caller sets the backflow fd, close
temporary file before exiting
Signed-off-by: David Barr <redacted>
Signed-off-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
---
contrib/svn-fe/svn-fe.txt | 5 +--
t/t9010-svn-fe.sh | 108 +++++++++++++++++++++++++++++++++++++++++++-
vcs-svn/fast_export.c | 109 ++++++++++++++++++++++++++++++++++++++++++++-
vcs-svn/fast_export.h | 3 +
vcs-svn/svndump.c | 13 ++++--
5 files changed, 227 insertions(+), 11 deletions(-)
@@ -32,9 +32,6 @@ Subversion's repository dump format is documented in full in Files in this format can be generated using the 'svnadmin dump' or 'svk admin dump' command.-Dumps produced with 'svnadmin dump --deltas' (dumpfile format v3)-are not supported.- OUTPUT FORMAT ------------- The fast-import format is documented by the git-fast-import(1)
@@ -836,6 +836,110 @@ test_expect_success PIPE 'deltas for typechange' 'test_cmpexpectactual'+test_expect_successPIPE'deltas need not consume the whole preimage''+reinit_git&&+cat>expect<<-\EOF&&+OBJID+:120000100644OBJIDOBJIDTpostimage+OBJID+:100644120000OBJIDOBJIDTpostimage+OBJID+:000000100644OBJIDOBJIDApostimage+EOF+echo"first preimage">expect.1&&+printftarget>expect.2&&+printflnk>expect.3&&+{+printf"SVNQ%b%b%b""QQ\017\001\017""\0217""first preimage\n"|+q_to_nul+}>delta.1&&+{+propertiessvn:special"*"&&+echoPROPS-END+}>symlink.props&&+{+printf"SVNQ%b%b%b""Q\002\013\004\012""\0201\001\001\0211""lnk target"|+q_to_nul+}>delta.2&&+{+printf"SVNQ%b%b""Q\004\003\004Q""\001Q\002\002"|+q_to_nul+}>delta.3&&+{+cat<<-\EOF&&+SVN-fs-dump-format-version:3++Revision-number:1+Prop-content-length:10+Content-length:10++PROPS-END++Node-path:postimage+Node-kind:file+Node-action:add+Text-delta:true+Prop-content-length:10+EOF+echoText-content-length:$(wc-c<delta.1)&&+echoContent-length:$((10+$(wc-c<delta.1)))&&+echo&&+echoPROPS-END&&+catdelta.1&&+cat<<-\EOF&&++Revision-number:2+Prop-content-length:10+Content-length:10++PROPS-END++Node-path:postimage+Node-kind:file+Node-action:change+Text-delta:true+EOF+echoProp-content-length:$(wc-c<symlink.props)&&+echoText-content-length:$(wc-c<delta.2)&&+echoContent-length:$(($(wc-c<symlink.props)+$(wc-c<delta.2)))&&+echo&&+catsymlink.props&&+catdelta.2&&+cat<<-\EOF&&++Revision-number:3+Prop-content-length:10+Content-length:10++PROPS-END++Node-path:postimage+Node-kind:file+Node-action:change+Text-delta:true+Prop-content-length:10+EOF+echoText-content-length:$(wc-c<delta.3)&&+echoContent-length:$((10+$(wc-c<delta.3)))&&+echo&&+echoPROPS-END&&+catdelta.3&&+echo+}>deltapartial.dump&&+try_dumpdeltapartial.dump&&+{+gitrev-listHEAD|+gitdiff-tree--root--stdin|+sed"s/$_x40/OBJID/g"+}>actual&&+test_cmpexpectactual&&+gitshowHEAD:postimage>actual.3&&+gitshowHEAD^:postimage>actual.2&&+gitshowHEAD^^:postimage>actual.1&&+test_cmpexpect.1actual.1&&+test_cmpexpect.2actual.2&&+test_cmpexpect.3actual.3+' test_expect_success'set up svn repo''svnconf=$PWD/svnconf&&
@@ -127,6 +150,73 @@ static void die_short_read(struct line_buffer *input)die("invalid dump: unexpected end of file");}+staticintends_with(constchar*s,size_tlen,constchar*suffix)+{+constsize_tsuffixlen=strlen(suffix);+if(len<suffixlen)+return0;+return!memcmp(s+len-suffixlen,suffix,suffixlen);+}++staticintparse_cat_response_line(constchar*header,off_t*len)+{+size_theaderlen=strlen(header);+constchar*type;+constchar*end;++if(ends_with(header,headerlen," missing"))+returnerror("cat-blob reports missing blob: %s",header);+type=memmem(header,headerlen," blob ",strlen(" blob "));+if(!type)+returnerror("cat-blob header has wrong object type: %s",header);+*len=strtoumax(type+strlen(" blob "),(char**)&end,10);+if(end==type+strlen(" blob "))+returnerror("cat-blob header does not contain length: %s",header);+if(*end)+returnerror("cat-blob header contains garbage after length: %s",header);+return0;+}++staticlongapply_delta(off_tlen,structline_buffer*input,+constchar*old_data,uint32_told_mode)+{+longret;+off_tpreimage_len=0;+structsliding_viewpreimage=SLIDING_VIEW_INIT(&report_buffer);+FILE*out;++if(init_postimage()||!(out=buffer_tmpfile_rewind(&postimage)))+die("cannot open temporary file for blob retrieval");+if(init_report_buffer(REPORT_FILENO))+die("cannot open fd 3 for feedback from fast-import");+if(old_data){+constchar*response;+printf("cat-blob %s\n",old_data);+fflush(stdout);+response=get_response_line();+if(parse_cat_response_line(response,&preimage_len))+die("invalid cat-blob response: %s",response);+}+if(old_mode==REPO_MODE_LNK){+strbuf_addstr(&preimage.buf,"link ");+preimage_len+=strlen("link ");+}+if(svndiff0_apply(input,len,&preimage,out))+die("cannot apply delta");+if(old_data){+/* Read the remainder of preimage and trailing newline. */+if(move_window(&preimage,preimage_len,1))+die("cannot seek to end of input");+if(preimage.buf.buf[0]!='\n')+die("missing newline after cat-blob response");+}+ret=buffer_tmpfile_prepare_to_read(&postimage);+if(ret<0)+die("cannot read temporary file for blob retrieval");+strbuf_release(&preimage.buf);+returnret;+}+voidfast_export_data(uint32_tmode,uint32_tlen,structline_buffer*input){if(mode==REPO_MODE_LNK){
@@ -15,6 +15,9 @@ void fast_export_begin_commit(uint32_t revision, const char *author,unsignedlongtimestamp);voidfast_export_end_commit(uint32_trevision);voidfast_export_data(uint32_tmode,uint32_tlen,structline_buffer*input);+voidfast_export_blob_delta(uint32_tmode,+uint32_told_mode,constchar*old_data,+uint32_tlen,structline_buffer*input);/* If there is no such file at that rev, returns -1, errno == ENOENT. */intfast_export_ls_rev(uint32_trev,constchar*path,
@@ -206,9 +206,7 @@ static void handle_node(void)*/staticconstchar*constempty_blob="::empty::";constchar*old_data=NULL;--if(node_ctx.text_delta)-die("text deltas not supported");+uint32_told_mode=REPO_MODE_BLB;if(node_ctx.action==NODEACT_DELETE){if(have_text||have_props||node_ctx.srcRev)
@@ -243,6 +241,7 @@ static void handle_node(void)if(mode!=REPO_MODE_DIR&&type==REPO_MODE_DIR)die("invalid dump: cannot modify a file into a directory");node_ctx.type=mode;+old_mode=mode;}elseif(node_ctx.action==NODEACT_ADD){if(type==REPO_MODE_DIR)old_data=NULL;
From: David Barr <hidden> Date: 2016-06-15 22:50:49
From: Jonathan Nieder <redacted>
The copyfrom_target instruction copies appends data that is already
present in the current output view to the end of output. (The offset
argument is relative to the beginning of output produced in the
current window.)
The region copied is allowed to run past the end of the existing
output. To support that case, copy one character at a time rather
than calling memcpy or memmove. This allows copyfrom_target to be
used once to repeat a string many times. For example:
COPYFROM_DATA 2
COPYFROM_OUTPUT 10, 0
DATA "ab"
would produce the output "ababababababababababab".
Signed-off-by: Jonathan Nieder <redacted>
Acked-by: Ramkumar Ramachandra <redacted>
Signed-off-by: David Barr <redacted>
---
t/t9011-svn-da.sh | 42 ++++++++++++++++++++++++++++++++++++++++++
vcs-svn/svndiff.c | 28 ++++++++++++++++++++++++++--
2 files changed, 68 insertions(+), 2 deletions(-)
@@ -170,4 +170,46 @@ test_expect_success 'catch attempt to copy missing data' 'test_must_failtest-svn-fe-dpreimagecopy.incomplete$len'+test_expect_success'copyfrom target to repeat data''+printffoofoo>expect&&+printf"SVNQ%b%b%s""QQ\006\004\003""\0203\0100\003Q""foo"|+q_to_nul>copytarget.repeat&&+len=$(wc-c<copytarget.repeat)&&+test-svn-fe-dpreimagecopytarget.repeat$len>actual&&+test_cmpexpectactual+'++test_expect_success'copyfrom target out of order''+printffoooof>expect&&+printf"SVNQ%b%b%s"\+"QQ\006\007\003""\0203\0101\002\0101\001\0101Q""foo"|+q_to_nul>copytarget.reverse&&+len=$(wc-c<copytarget.reverse)&&+test-svn-fe-dpreimagecopytarget.reverse$len>actual&&+test_cmpexpectactual+'++test_expect_success'catch copyfrom future''+printf"SVNQ%b%b%s""QQ\004\004\003""\0202\0101\002\0201""XYZ"|+q_to_nul>copytarget.infuture&&+len=$(wc-c<copytarget.infuture)&&+test_must_failtest-svn-fe-dpreimagecopytarget.infuture$len+'++test_expect_success'copy to sustain''+printfXYXYXYXYXYXZ>expect&&+printf"SVNQ%b%b%s""QQ\014\004\003""\0202\0111Q\0201""XYZ"|+q_to_nul>copytarget.sustain&&+len=$(wc-c<copytarget.sustain)&&+test-svn-fe-dpreimagecopytarget.sustain$len>actual&&+test_cmpexpectactual+'++test_expect_success'catch copy that overflows''+printf"SVNQ%b%b%s""QQ\003\003\001""\0201\0177Q"X|+q_to_nul>copytarget.overflow&&+len=$(wc-c<copytarget.overflow)&&+test_must_failtest-svn-fe-dpreimagecopytarget.overflow$len+'+ test_done
Side note: it's often not clear what should go in the "prepare for next
user" routine and what should go in the "shutting down for good". I
suppose these should use strbuf_reset and the memory would be finally
freed in svndump_reset? Does it make sense to have two distinct
routines like this without a user to demonstrate the trade-offs?
Except as noted above,
Reviewed-by: Jonathan Nieder <redacted>
Thanks; I like where this is going.
Obviously good, both because it means we can use one round-trip
to get the same information that previously took two and because
the API just makes more sense. Thanks.
Crazy idea: to make it visible at a glance when the numbers are wrong,
one can do:
switch (key_len + 1) {
case sizeof("svn:log"):
if (memcmp(key, "svn:log", strlen("svn:log")))
break;
This only makes the redundancy more obvious, of course. It could
be reduced a little with something like
static int prefixcmp_len(const char *str, size_t str_len,
const char *prefix, size_t prefix_len)
{
if (prefix_len > str_len)
return 1;
return memcmp(str, prefix, prefix_len);
}
but that's probably not worth the cognitive load.
[...]
- } else if (key == keys.svn_executable || key == keys.svn_special) {
+ break;
+ case 14:
+ if (memcmp(key, "svn:executable", 14))
+ break;
+ case 11:
+ if (key_len == 11 && memcmp(key, "svn:special", 11))
+ break;
Maybe, to avoid an unnecessary /* fall through */:
case sizeof("svn:executable"):
case sizeof("svn:special"):
if (key_len == strlen("svn:executable") &&
memcmp(key, "svn:executable", strlen(...)))
break;
if (key_len == strlen("svn:special") &&
memcmp(key, "svn:special", strlen("svn:special")))
break;
Probably warrants a comment:
/* the longest key we pay attention to is "<whatever>" */
quoted hunk
const char *t;
/*
* NEEDSWORK: to support simple mode changes like
@@ -175,16 +185,20 @@ static void read_props(void) switch (type) { case 'K':- key = pool_intern(val);- continue; case 'D':- key = pool_intern(val);+ if (len < sizeof(key))+ memcpy(key, val, len + 1);
What happens on I/O error, when val is NULL? How about early EOF
or malformed input, when strlen(val) < len?
Some tests would also be a comfort.
I'm not so happy with the table of (at first glance) magic-seeming
numbers and the error handling looks a little tricky but aside from
those details this seems like a reasonable way to avoid some
complication without sacrificing speed.
Speaking of which, any hints for people who want to time this patch
(and other patches in the series)?
Thanks.
Jonathan
Same comments as the previous patch apply here.
Might make sense to split out the loop body (or at least the giant
switch statement) as a separate function for easier contemplation.
[...]
- } else if (key == keys.content_length) {
+ break;
+ case 14:
+ if (memcmp(t, "Content-length", 14))
+ continue;
len = atoi(val);
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:50:49
David Barr wrote:
[Subject: vcs-svn: factor out usage of string_pool]
This actually means something like: use strbufs and strings instead of
interned strings for values of rev, dump, and node fields that happen
to be strings. After this change, there are no more users of the
string-pool library left.
Side note: should the default timestamp really be the epoch? I'd
rather the default timestamp be the timestamp of the parent revision,
to make out-of-order dates a little less likely.
From: David Barr <hidden> Date: 2016-06-15 22:50:49
This eliminates one more dependency on string_pool.
Signed-off-by: David Barr <redacted>
Signed-off-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
Reviewed-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
---
vcs-svn/svndump.c | 110 +++++++++++++++++++++++++++++------------------------
1 files changed, 60 insertions(+), 50 deletions(-)
@@ -51,14 +51,6 @@ static struct {uint32_tversion,uuid,url;}dump_ctx;-staticstruct{-uint32_tuuid,revision_number,node_path,node_kind,node_action,-node_copyfrom_path,node_copyfrom_rev,text_content_length,-prop_content_length,content_length,svn_fs_dump_format_version,-/* version 3 format */-text_delta,prop_delta;-}keys;-staticvoidreset_node_ctx(char*fname){node_ctx.type=0;
@@ -89,24 +81,6 @@ static void reset_dump_ctx(uint32_t url)dump_ctx.uuid=~0;}-staticvoidinit_keys(void)-{-keys.uuid=pool_intern("UUID");-keys.revision_number=pool_intern("Revision-number");-keys.node_path=pool_intern("Node-path");-keys.node_kind=pool_intern("Node-kind");-keys.node_action=pool_intern("Node-action");-keys.node_copyfrom_path=pool_intern("Node-copyfrom-path");-keys.node_copyfrom_rev=pool_intern("Node-copyfrom-rev");-keys.text_content_length=pool_intern("Text-content-length");-keys.prop_content_length=pool_intern("Prop-content-length");-keys.content_length=pool_intern("Content-length");-keys.svn_fs_dump_format_version=pool_intern("SVN-fs-dump-format-version");-/* version 3 format (Subversion 1.1.0) */-keys.text_delta=pool_intern("Text-delta");-keys.prop_delta=pool_intern("Prop-delta");-}-/* Compare string to literal of equal length; must be guarded by length test. */#define constcmp(s, ref) memcmp((s), (ref), sizeof(ref) - 1)
From: David Barr <hidden> Date: 2016-06-15 22:50:49
In the spirit of the last two changes:
Switch on length and use constcmp for parsing headers with restricted values.
Signed-off-by: David Barr <redacted>
---
vcs-svn/svndump.c | 38 +++++++++++++++++++++++++++++---------
1 files changed, 29 insertions(+), 9 deletions(-)
From: David Barr <hidden> Date: 2016-06-15 22:50:49
This eliminates one more dependency on string_pool.
Signed-off-by: David Barr <redacted>
Signed-off-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
Reviewed-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
---
vcs-svn/svndump.c | 64 +++++++++++++++++++++++++++++++++++++---------------
1 files changed, 45 insertions(+), 19 deletions(-)
From: David Barr <hidden> Date: 2016-06-15 22:50:49
In the spirit of the last two changes:
Switch on length and use constcmp for parsing headers with restricted values.
Signed-off-by: David Barr <redacted>
---
Silly me, the last version didn't even compile.
vcs-svn/svndump.c | 39 +++++++++++++++++++++++++++++----------
1 files changed, 29 insertions(+), 10 deletions(-)
From: David Barr <hidden> Date: 2016-06-15 22:50:50
In the spirit of the last two changes:
Switch on length and use constcmp for parsing headers with restricted values.
Signed-off-by: David Barr <redacted>
Reviewed-by: Jonathan Nieder <redacted>
---
vcs-svn/svndump.c | 40 ++++++++++++++++++++++++++++++----------
1 files changed, 30 insertions(+), 10 deletions(-)
@@ -236,8 +236,7 @@ static void handle_node(void)old_data=NULL;}elseif(node_ctx.action==NODEACT_CHANGE){uint32_tmode;-old_data=repo_read_path(node_ctx.dst.buf);-mode=repo_read_mode(node_ctx.dst.buf);+old_data=repo_read_path(node_ctx.dst.buf,&mode);if(mode==REPO_MODE_DIR&&type!=REPO_MODE_DIR)die("invalid dump: cannot modify a directory into a file");if(mode!=REPO_MODE_DIR&&type==REPO_MODE_DIR)
From: David Barr <hidden> Date: 2016-06-15 22:50:50
This is a small step towards removing dependence on obj_pool.h
Signed-off-by: David Barr <redacted>
Reviewed-by: Jonathan Nieder <redacted>
---
vcs-svn/svndump.c | 26 +++++++-------------------
1 files changed, 7 insertions(+), 19 deletions(-)
From: David Barr <hidden> Date: 2016-06-15 22:50:50
This eliminates one more dependency on string_pool.
Signed-off-by: David Barr <redacted>
Signed-off-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
Reviewed-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
---
vcs-svn/svndump.c | 57 +++++++++++++++++++++++++++++++++++-----------------
1 files changed, 38 insertions(+), 19 deletions(-)
From: David Barr <hidden> Date: 2016-06-15 22:50:50
Now that there is no internal representation of the repo,
it is not necessary to tokenise paths.
Use strbuf instead and bypass string_pool.
Also, since we now treat paths in their entirety,
only quote when necessary.
Signed-off-by: David Barr <redacted>
Signed-off-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
Reviewed-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
---
vcs-svn/fast_export.c | 47 ++++++++++++++++++++++++-----------------------
vcs-svn/fast_export.h | 9 ++++-----
vcs-svn/repo_tree.c | 20 ++++++++++----------
vcs-svn/repo_tree.h | 13 +++++--------
vcs-svn/svndump.c | 37 ++++++++++++++++++++-----------------
5 files changed, 63 insertions(+), 63 deletions(-)
@@ -8,18 +8,17 @@ void fast_export_init(int fd);voidfast_export_deinit(void);voidfast_export_reset(void);-voidfast_export_delete(uint32_tdepth,constuint32_t*path);-voidfast_export_modify(uint32_tdepth,constuint32_t*path,-uint32_tmode,constchar*dataref);+voidfast_export_delete(constchar*path);+voidfast_export_modify(constchar*path,uint32_tmode,constchar*dataref);voidfast_export_begin_commit(uint32_trevision,uint32_tauthor,char*log,uint32_tuuid,uint32_turl,unsignedlongtimestamp);voidfast_export_end_commit(uint32_trevision);voidfast_export_data(uint32_tmode,uint32_tlen,structline_buffer*input);/* If there is no such file at that rev, returns -1, errno == ENOENT. */-intfast_export_ls_rev(uint32_trev,uint32_tdepth,constuint32_t*path,+intfast_export_ls_rev(uint32_trev,constchar*path,uint32_t*mode_out,structstrbuf*dataref_out);-intfast_export_ls(uint32_tdepth,constuint32_t*path,+intfast_export_ls(constchar*path,uint32_t*mode_out,structstrbuf*dataref_out);#endif
@@ -211,14 +213,14 @@ static void handle_node(void)if(have_text||have_props||node_ctx.srcRev)die("invalid dump: deletion node has ""copyfrom info, text, or properties");-returnrepo_delete(node_ctx.dst);+returnrepo_delete(node_ctx.dst.buf);}if(node_ctx.action==NODEACT_REPLACE){-repo_delete(node_ctx.dst);+repo_delete(node_ctx.dst.buf);node_ctx.action=NODEACT_ADD;}if(node_ctx.srcRev){-repo_copy(node_ctx.srcRev,node_ctx.src,node_ctx.dst);+repo_copy(node_ctx.srcRev,node_ctx.src.buf,node_ctx.dst.buf);if(node_ctx.action==NODEACT_ADD)node_ctx.action=NODEACT_CHANGE;}
@@ -228,14 +230,14 @@ static void handle_node(void)/**Findoldcontent(old_data)anddecideonthenewmode.*/-if(node_ctx.action==NODEACT_CHANGE&&!~*node_ctx.dst){+if(node_ctx.action==NODEACT_CHANGE&&!*node_ctx.dst.buf){if(type!=REPO_MODE_DIR)die("invalid dump: root of tree is not a regular file");old_data=NULL;}elseif(node_ctx.action==NODEACT_CHANGE){uint32_tmode;-old_data=repo_read_path(node_ctx.dst);-mode=repo_read_mode(node_ctx.dst);+old_data=repo_read_path(node_ctx.dst.buf);+mode=repo_read_mode(node_ctx.dst.buf);if(mode==REPO_MODE_DIR&&type!=REPO_MODE_DIR)die("invalid dump: cannot modify a directory into a file");if(mode!=REPO_MODE_DIR&&type==REPO_MODE_DIR)
@@ -272,12 +274,10 @@ static void handle_node(void)/* For the fast_export_* functions, NULL means empty. */old_data=NULL;if(!have_text){-fast_export_modify(REPO_MAX_PATH_DEPTH,node_ctx.dst,-node_ctx.type,old_data);+fast_export_modify(node_ctx.dst.buf,node_ctx.type,old_data);return;}-fast_export_modify(REPO_MAX_PATH_DEPTH,node_ctx.dst,-node_ctx.type,"inline");+fast_export_modify(node_ctx.dst.buf,node_ctx.type,"inline");fast_export_data(node_ctx.type,node_ctx.textLength,&input);}
From: David Barr <hidden> Date: 2016-06-15 22:50:50
This eliminates one more dependency on string_pool.
Signed-off-by: David Barr <redacted>
Signed-off-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
Reviewed-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
---
vcs-svn/svndump.c | 110 +++++++++++++++++++++++++++++------------------------
1 files changed, 60 insertions(+), 50 deletions(-)
@@ -51,14 +51,6 @@ static struct {uint32_tversion,uuid,url;}dump_ctx;-staticstruct{-uint32_tuuid,revision_number,node_path,node_kind,node_action,-node_copyfrom_path,node_copyfrom_rev,text_content_length,-prop_content_length,content_length,svn_fs_dump_format_version,-/* version 3 format */-text_delta,prop_delta;-}keys;-staticvoidreset_node_ctx(char*fname){node_ctx.type=0;
@@ -89,24 +81,6 @@ static void reset_dump_ctx(uint32_t url)dump_ctx.uuid=~0;}-staticvoidinit_keys(void)-{-keys.uuid=pool_intern("UUID");-keys.revision_number=pool_intern("Revision-number");-keys.node_path=pool_intern("Node-path");-keys.node_kind=pool_intern("Node-kind");-keys.node_action=pool_intern("Node-action");-keys.node_copyfrom_path=pool_intern("Node-copyfrom-path");-keys.node_copyfrom_rev=pool_intern("Node-copyfrom-rev");-keys.text_content_length=pool_intern("Text-content-length");-keys.prop_content_length=pool_intern("Prop-content-length");-keys.content_length=pool_intern("Content-length");-keys.svn_fs_dump_format_version=pool_intern("SVN-fs-dump-format-version");-/* version 3 format (Subversion 1.1.0) */-keys.text_delta=pool_intern("Text-delta");-keys.prop_delta=pool_intern("Prop-delta");-}-/* Compare string to literal of equal length; must be guarded by length test. */#define constcmp(s, ref) memcmp((s), (ref), sizeof(ref) - 1)
From: David Barr <hidden> Date: 2016-06-15 22:50:50
This is a small optimisation (4% reduction in user time) but is the largest
artifact within the parsing portion of svndump.c
Signed-off-by: David Barr <redacted>
Signed-off-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
Reviewed-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
---
vcs-svn/svndump.c | 4 +++-
1 files changed, 3 insertions(+), 1 deletions(-)
@@ -1,43 +0,0 @@-string_pool API-===============--The string_pool API provides facilities for replacing strings-with integer keys that can be more easily compared and stored.-The facilities are designed so that one could teach Git without-too much trouble to store the information needed for these keys to-remain valid over multiple executions.--Functions------------pool_intern::- Include a string in the string pool and get its key.- If that string is already in the pool, retrieves its- existing key.--pool_fetch::- Retrieve the string associated to a given key.--pool_tok_r::- Extract the key of the next token from a string.- Interface mimics strtok_r.--pool_print_seq::- Print a sequence of strings named by key to a file, using the- specified delimiter to separate them.-- If NULL (key ~0) appears in the sequence, the sequence ends- early.--pool_tok_seq::- Split a string into tokens, storing the keys of segments- into a caller-provided array.-- Unless sz is 0, the array will always be ~0-terminated.- If there is not enough room for all the tokens, the- array holds as many tokens as fit in the entries before- the terminating ~0. Return value is the index after the- last token, or sz if the tokens did not fit.--pool_reset::- Deallocate storage for the string pool.
From: David Barr <hidden> Date: 2016-06-15 22:50:50
Thanks Jonathan for reviewing the series and for suggesting
a good compromise between readability and performance.
The first patch of the last version has been split into two.
Patch 6 follows the spirit of patches 4 and 5, for a consistent
approach to switching on constant strings.
.gitignore | 3 -
Makefile | 13 +--
t/t0080-vcs-svn.sh | 117 ------------------
test-obj-pool.c | 116 ------------------
test-string-pool.c | 31 -----
test-treap.c | 70 -----------
vcs-svn/LICENSE | 3 -
vcs-svn/fast_export.c | 64 +++++-----
vcs-svn/fast_export.h | 14 +-
vcs-svn/obj_pool.h | 61 ---------
vcs-svn/repo_tree.c | 36 ++----
vcs-svn/repo_tree.h | 12 +--
vcs-svn/string_pool.c | 113 -----------------
vcs-svn/string_pool.h | 12 --
vcs-svn/string_pool.txt | 43 -------
vcs-svn/svndump.c | 309 ++++++++++++++++++++++++++++-------------------
vcs-svn/trp.h | 237 ------------------------------------
vcs-svn/trp.txt | 109 -----------------
18 files changed, 237 insertions(+), 1126 deletions(-)
From: David Barr <hidden> Date: 2016-06-15 22:50:50
That is, use strbufs and strings instead of interned
strings for values of rev, dump, and node fields that
happen to be strings. After this change, there are
no more users of the string-pool library left.
There is a small functional change inlined: test for
emtpy rather than NULL when falling back to defaults
for commit metadata.
Signed-off-by: David Barr <redacted>
Signed-off-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
Reviewed-by: Jonathan Nieder <redacted>
Signed-off-by: David Barr <redacted>
---
vcs-svn/fast_export.c | 17 +++++++----------
vcs-svn/fast_export.h | 5 +++--
vcs-svn/svndump.c | 46 ++++++++++++++++++++++++++++++----------------
3 files changed, 40 insertions(+), 28 deletions(-)
@@ -71,14 +71,16 @@ static void reset_rev_ctx(uint32_t revision)rev_ctx.revision=revision;rev_ctx.timestamp=0;strbuf_reset(&rev_ctx.log);-rev_ctx.author=~0;+strbuf_reset(&rev_ctx.author);}-staticvoidreset_dump_ctx(uint32_turl)+staticvoidreset_dump_ctx(constchar*url){-dump_ctx.url=url;+strbuf_reset(&dump_ctx.url);+if(url)+strbuf_addstr(&dump_ctx.url,url);dump_ctx.version=1;-dump_ctx.uuid=~0;+strbuf_reset(&dump_ctx.uuid);}/* Compare string to literal of equal length; must be guarded by length test. */
@@ -1,9 +1,6 @@ Copyright (C) 2010 David Barr <david.barr@cordelta.com>. All rights reserved.-Copyright (C) 2008 Jason Evans <jasone@canonware.com>.-All rights reserved.- Copyright (C) 2005 Stefan Hegny, hydrografix Consulting GmbH, Frankfurt/Main, Germany and others, see http://svn2cc.sarovar.org
@@ -1,109 +0,0 @@-Motivation-==========--Treaps provide a memory-efficient binary search tree structure.-Insertion/deletion/search are about as about as fast in the average-case as red-black trees and the chances of worst-case behavior are-vanishingly small, thanks to (pseudo-)randomness. The bad worst-case-behavior is a small price to pay, given that treaps are much simpler-to implement.--API-===--The trp API generates a data structure and functions to handle a-large growing set of objects stored in a pool.--The caller:--. Specifies parameters for the generated functions with the- trp_gen(static, foo_, ...) macro.--. Allocates a `struct trp_root` variable and sets it to {~0}.--. Adds new nodes to the set using `foo_insert`. Any pointers- to existing nodes cannot be relied upon any more, so the caller- might retrieve them anew with `foo_pointer`.--. Can find a specific item in the set using `foo_search`.--. Can iterate over items in the set using `foo_first` and `foo_next`.--. Can remove an item from the set using `foo_remove`.--Example:-------struct ex_node {- const char *s;- struct trp_node ex_link;-};-static struct trp_root ex_base = {~0};-obj_pool_gen(ex, struct ex_node, 4096);-trp_gen(static, ex_, struct ex_node, ex_link, ex, strcmp)-struct ex_node *item;--item = ex_pointer(ex_alloc(1));-item->s = "hello";-ex_insert(&ex_base, item);-item = ex_pointer(ex_alloc(1));-item->s = "goodbye";-ex_insert(&ex_base, item);-for (item = ex_first(&ex_base); item; item = ex_next(&ex_base, item))- printf("%s\n", item->s);-------Functions------------trp_gen(attr, foo_, node_type, link_field, pool, cmp)::-- Generate a type-specific treap implementation.-+-. The storage class for generated functions will be 'attr' (e.g., `static`).-. Generated function names are prefixed with 'foo_' (e.g., `treap_`).-. Treap nodes will be of type 'node_type' (e.g., `struct treap_node`).- This type must be a struct with at least one `struct trp_node` field- to point to its children.-. The field used to access child nodes will be 'link_field'.-. All treap nodes must lie in the 'pool' object pool.-. Treap nodes must be totally ordered by the 'cmp' relation, with the- following prototype:-+-int (*cmp)(node_type \*a, node_type \*b)-+-and returning a value less than, equal to, or greater than zero-according to the result of comparison.--node_type {asterisk}foo_insert(struct trp_root *treap, node_type \*node)::-- Insert node into treap. If inserted multiple times,- a node will appear in the treap multiple times.-+-The return value is the address of the node within the treap,-which might differ from `node` if `pool_alloc` had to call-`realloc` to expand the pool.--void foo_remove(struct trp_root *treap, node_type \*node)::-- Remove node from treap. Caller must ensure node is- present in treap before using this function.--node_type *foo_search(struct trp_root \*treap, node_type \*key)::-- Search for a node that matches key. If no match is found,- result is NULL.--node_type *foo_nsearch(struct trp_root \*treap, node_type \*key)::-- Like `foo_search`, but if if the key is missing return what- would be key's successor, were key in treap (NULL if no- successor).--node_type *foo_first(struct trp_root \*treap)::-- Find the first item from the treap, in sorted order.--node_type *foo_next(struct trp_root \*treap, node_type \*node)::-- Find the next item.
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:50:51
(culled cc list)
Hi Junio,
Please pull
git://repo.or.cz/git/jrn.git svn-fe
to get the following changes on top of master.
These are the patches from David's recent code-purge series that do
not require incremental import support. I'd like to push out
incremental import support soon, too, but since that makes svn-fe
require feedback from fast-import as it runs, it would be nice to
provide some simple wrapper script to set everything up at the same
time to avoid inconveniencing users too much.
The main impact of the patches currently in svn-fe should be to
improve error handling a little.
David Barr wrote:
Patch 6 follows the spirit of patches 4 and 5, for a consistent
approach to switching on constant strings.
I've skipped this one and applied the rest. You can see the result
in the svn-fe-pu branch.
Thoughts, suggestions, improvements welcome as always.
David Barr (5):
vcs-svn: use strbuf for revision log
vcs-svn: use strbuf for author, UUID, and URL
vcs-svn: implement perfect hash for node-prop keys
vcs-svn: implement perfect hash for top-level keys
vcs-svn: use strchr to find RFC822 delimiter
Jonathan Nieder (9):
vcs-svn: introduce repo_read_path to check the content at a path
vcs-svn: handle_node: use repo_read_path
vcs-svn: simplify repo_modify_path and repo_copy
vcs-svn: allow input errors to be detected promptly
vcs-svn: improve support for reading large files
vcs-svn: make buffer_skip_bytes return length read
vcs-svn: make buffer_copy_bytes return length read
vcs-svn: improve reporting of input errors
Merge branch 'db/length-as-hash' into svn-fe
vcs-svn/fast_export.c | 27 +++--
vcs-svn/fast_export.h | 5 +-
vcs-svn/line_buffer.c | 36 ++++---
vcs-svn/line_buffer.h | 6 +-
vcs-svn/line_buffer.txt | 3 +-
vcs-svn/repo_tree.c | 43 ++++---
vcs-svn/repo_tree.h | 10 +-
vcs-svn/svndump.c | 307 +++++++++++++++++++++++++++++------------------
8 files changed, 265 insertions(+), 172 deletions(-)
From: Junio C Hamano <hidden> Date: 2016-06-15 22:50:51
Jonathan Nieder [off-list ref] writes:
Please pull
git://repo.or.cz/git/jrn.git svn-fe
to get the following changes on top of master.
Done.
I only gave a cursory look at "git diff ORIG_HEAD" output immediately
after pulling, but I found that the majority of lines deleted were of
questionable style and the added ones looked more like normal C ;-)
Except for
switch (keylen + 1) {
case sizeof("constant string"):
...
break;
case sizeof("another constant string"):
...
}
which looked a bit unusual. But mistakes in this construct can be easily
caught by the compiler that would notice duplicated case labels, so it
probably is not so brittle as it first looks.
By the way, I've been getting annoyed by these three "sleep 100" getting
stuck in t0081 and spending their sweet timeout while running my tests
(prove is on, "make test </dev/null" to forbid it from reading my stdin).
I see attempts to kill them early with "kill $!" but apparently they are
not working. Can you take a look at it?
From: David Barr <hidden> Date: 2016-06-15 22:50:51
Hi,
Junio C Hamano wrote:
Jonathan Nieder [off-list ref] writes:
quoted
Please pull
git://repo.or.cz/git/jrn.git svn-fe
to get the following changes on top of master.
Done.
I only gave a cursory look at "git diff ORIG_HEAD" output immediately
after pulling, but I found that the majority of lines deleted were of
questionable style and the added ones looked more like normal C ;-)
Except for
switch (keylen + 1) {
case sizeof("constant string"):
...
break;
case sizeof("another constant string"):
...
}
which looked a bit unusual. But mistakes in this construct can be easily
caught by the compiler that would notice duplicated case labels, so it
probably is not so brittle as it first looks.
Agreed, it is quite an odd construct but it took quite a bit of refinement to
arrive there. It is a compromise between moderately readable and
reasonably fast. I did play around with perfect hash generators prior to
settling on this approach. Ditto re compile time checking.
Jonathan, I suppose I should set up a performance test for these patches
because my first thought was "I wonder what these buy us on their own?"
--
David Barr.
From: David Barr <hidden> Date: 2016-06-15 22:50:52
An excessive constraint was introduced in c9d1c8ba; when reading
svn props, it is permissible for both keys and values to contain
nul characters. Thus the test `strlen(val) != len` may fail on
such properties. This caused svn-fe to die early whilst handling
revision 59151 of the ASF repository.
---
vcs-svn/svndump.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:50:52
Hi David,
David Barr wrote:
An excessive constraint was introduced in c9d1c8ba; when reading
svn props, it is permissible for both keys and values to contain
nul characters.
Yes, that will work.
buffer_read_string returns a '\0'-terminated string and on early EOF,
the part after the end-of-file will be gibberish (and probably
uninitialized). But it lives in an 1000-char buffer and errors out
when it doesn't fit, so at least with the fix it wouldn't crash.
Sorry for the breakage. I suppose a test like the following would
catch future problems of this kind?
Signed-off-by: Jonathan Nieder <redacted>
---
t/t9010-svn-fe.sh | 79 +++++++++++++++++++++++++++++++++++++++++++++++++++++
1 files changed, 79 insertions(+), 0 deletions(-)
@@ -370,6 +370,85 @@ test_expect_failure 'change file mode but keep old content' 'test_cmphelloactual.target'+test_expect_failure'null bytes''+# Caveat: svnadmin 1.6.16 (r1073529) truncates at \0 in the+# svn:specialQnotreally example.+reinit_git&&+cat>expect<<-\EOF&&+OBJID+:100644100644OBJIDOBJIDMgreeting+OBJID+:000000100644OBJIDOBJIDAgreeting+EOF+printf"%s\n""something with a null byte (Q)"|+q_to_nul>expect.message&&+printf"%s\n""helQo"|+q_to_nul>expect.hello1&&+printf"%s\n""link hello">expect.hello2&&+{+propertiessvn:log"something with a null byte (Q)"&&+echoPROPS-END+}|+q_to_nul>props&&+{+q_to_nul<<-\EOF&&+SVN-fs-dump-format-version:3++Revision-number:1+Prop-content-length:10+Content-length:10++PROPS-END++Node-path:greeting+Node-kind:file+Node-action:add+Prop-content-length:10+Text-content-length:6+Content-length:16++PROPS-END+helQo++Revision-number:2+EOF+echoProp-content-length:$(wc-c<props)&&+echoContent-length:$(wc-c<props)&&+echo&&+catprops&&+q_to_nul<<-\EOF++Node-path:greeting+Node-kind:file+Node-action:change+Prop-content-length:43+Text-content-length:11+Content-length:54++K21+svn:specialQnotreally+V1+*+PROPS-END+linkhello+EOF+}>8bitclean.dump&&+test-svn-fe8bitclean.dump>stream&&+gitfast-import<stream&&+{+gitrev-listHEAD|+gitdiff-tree--root--stdin|+sed"s/$_x40/OBJID/g"+}>actual&&+gitdiff-tree--always-s--format=%sHEAD>actual.message&&+gitcat-fileblobHEAD^:greeting>actual.hello1&&+gitcat-fileblobHEAD:greeting>actual.hello2&&+test_cmpexpectactual&&+test_cmpexpect.messageactual.message&&+test_cmpexpect.hello1actual.hello1&&+test_cmpexpect.hello2actual.hello2+'+ test_expect_success'change file mode and reiterate content''reinit_git&&cat>expect<<-\EOF&&
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:50:52
David Barr wrote:
it is permissible for both keys and values to contain
nul characters.
You're right --- it's a regression to error out, though we never did
support it all that well. How about this?
This doesn't take care of preserving embedded null bytes in the author
name. That can come another day, I suppose.
Jonathan Nieder (4):
vcs-svn: make reading of properties binary-safe
vcs-svn: remove buffer_read_string
vcs-svn: avoid unnecessary copying of log message and author
vcs-svn: handle log message with embedded null bytes
t/t0081-line-buffer.sh | 35 ++++++----------
t/t9010-svn-fe.sh | 104 +++++++++++++++++++++++++++++++++++++++++++++++
test-line-buffer.c | 6 ---
vcs-svn/fast_export.c | 12 +++--
vcs-svn/fast_export.h | 9 ++--
vcs-svn/line_buffer.c | 8 ----
vcs-svn/line_buffer.h | 4 +-
vcs-svn/line_buffer.txt | 12 +----
vcs-svn/repo_tree.c | 5 +-
vcs-svn/repo_tree.h | 4 +-
vcs-svn/svndump.c | 42 +++++++++----------
11 files changed, 157 insertions(+), 84 deletions(-)
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:50:52
A caller to buffer_read_string cannot easily tell the difference
between the string "foo" followed by an early end of file and the
string "foo\0bar\0baz". In a half-hearted attempt to catch early EOF,
c9d1c8ba (2010-12-28) introduced a safety strlen(val) == len for
property keys and values, to at least keep svn-fe from reading
uninitialized data when a property list ends early due to EOF.
But it is permissible for both keys and values to contain null
characters, so in handling revision 59151 of the ASF repository svn-fe
encounters a null byte and produces the following message:
fatal: invalid dump: unexpected end of file
Fix it by using buffer_read_binary to read to a strbuf (and keep track
of the actual length read). Most consumers of properties still use
C-style strings, so in practice we still can't use an author or log
message with embedded nuls, but at least this way svn-fe won't error
out.
Reported-by: David Barr <redacted>
Signed-off-by: Jonathan Nieder <redacted>
---
t/t9010-svn-fe.sh | 27 +++++++++++++++++++++++++++
vcs-svn/svndump.c | 24 ++++++++++--------------
2 files changed, 37 insertions(+), 14 deletions(-)
@@ -370,6 +370,33 @@ test_expect_failure 'change file mode but keep old content' 'test_cmphelloactual.target'+test_expect_success'null byte in property value''+reinit_git&&+echo"commit message">expect.message&&+{+properties\+unimportant"something with a null byte (Q)"\+svn:log"commit message"&&+echoPROPS-END+}|+q_to_nul>props&&+{+cat<<-\EOF&&+SVN-fs-dump-format-version:3++Revision-number:1+EOF+echoProp-content-length:$(wc-c<props)&&+echoContent-length:$(wc-c<props)&&+echo&&+catprops+}>nullprop.dump&&+test-svn-fenullprop.dump>stream&&+gitfast-import<stream&&+gitdiff-tree--always-s--format=%sHEAD>actual.message&&+test_cmpexpect.messageactual.message+'+ test_expect_success'change file mode and reiterate content''reinit_git&&cat>expect<<-\EOF&&
@@ -179,22 +180,17 @@ static void read_props(void)if(ch==EOF)die_short_read();if(ch!='\n')-die("invalid dump: expected newline after %s",val);+die("invalid dump: expected newline after %s",val.buf);switch(type){case'K':+strbuf_swap(&key,&val);+continue;case'D':-strbuf_reset(&key);-if(val)-strbuf_add(&key,val,len);-if(type=='K')-continue;-assert(type=='D');-val=NULL;-len=0;-/* fall through */+handle_property(&val,NULL,0,&type_set);+continue;case'V':-handle_property(&key,val,len,&type_set);+handle_property(&key,val.buf,len,&type_set);strbuf_reset(&key);continue;default:
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:50:52
All previous users of buffer_read_string have already been converted
to use the more intuitive buffer_read_binary, so remove the old API to
avoid some confusion.
Signed-off-by: Jonathan Nieder <redacted>
---
t/t0081-line-buffer.sh | 35 +++++++++++++----------------------
test-line-buffer.c | 6 ------
vcs-svn/line_buffer.c | 8 --------
vcs-svn/line_buffer.h | 4 +---
vcs-svn/line_buffer.txt | 12 +++---------
5 files changed, 17 insertions(+), 48 deletions(-)
@@ -16,8 +16,8 @@ The calling program: - initializes a `struct line_buffer` to LINE_BUFFER_INIT - specifies a file to read with `buffer_init`- - processes input with `buffer_read_line`, `buffer_read_string`,- `buffer_skip_bytes`, and `buffer_copy_bytes`+ - processes input with `buffer_read_line`, `buffer_skip_bytes`,+ and `buffer_copy_bytes` - closes the file with `buffer_deinit`, perhaps to start over and read another file.
@@ -37,7 +37,7 @@ the calling program. A program the temporary file - declares writing is over with `buffer_tmpfile_prepare_to_read` - can re-read what was written with `buffer_read_line`,- `buffer_read_string`, and so on+ `buffer_copy_bytes`, and so on - can reuse the temporary file by calling `buffer_tmpfile_rewind` again - removes the temporary file with `buffer_deinit`, perhaps to
@@ -64,12 +64,6 @@ Functions Read a line and strip off the trailing newline. On failure or end of file, returns NULL.-`buffer_read_string`::- Read `len` characters of input or up to the end of the- file, whichever comes first. Returns NULL on error.- Returns whatever characters were read (possibly "")- for end of file.- `buffer_copy_bytes`:: Read `len` bytes of input and dump them to the standard output stream. Returns early for error or end of file.
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:50:52
Use strbuf_swap when storing the svn:log and svn:author properties, so
pointers to rather than the contents of buffers get copied. The main
effect should be to make the code a little easier to read.
Signed-off-by: Jonathan Nieder <redacted>
---
vcs-svn/svndump.c | 20 ++++++++++----------
1 files changed, 10 insertions(+), 10 deletions(-)
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:50:54
Hi,
Junio C Hamano wrote:
Done.
Turns out that introduced a regression (in handling of properties with
embedded NULs; thanks to David for catching and debugging it). Could
you pull
git://repo.or.cz/git/jrn.git svn-fe
for a fix and some other improvements in the area?
Sorry for the trouble,
Jonathan Nieder (4):
vcs-svn: make reading of properties binary-safe
vcs-svn: remove buffer_read_string
vcs-svn: avoid unnecessary copying of log message and author
vcs-svn: handle log message with embedded NUL
t/t0081-line-buffer.sh | 35 ++++++----------
t/t9010-svn-fe.sh | 104 +++++++++++++++++++++++++++++++++++++++++++++++
test-line-buffer.c | 6 ---
vcs-svn/fast_export.c | 12 +++--
vcs-svn/fast_export.h | 7 ++-
vcs-svn/line_buffer.c | 8 ----
vcs-svn/line_buffer.h | 4 +-
vcs-svn/line_buffer.txt | 12 +----
vcs-svn/repo_tree.c | 5 +-
vcs-svn/repo_tree.h | 4 +-
vcs-svn/svndump.c | 42 +++++++++----------
11 files changed, 156 insertions(+), 83 deletions(-)
Apparently this change is from in an evil merge. Yikes.
Anyway, I think the original patch was something like the following.
Would you mind if the parameter and return value go back to being of
type size_t (to avoid a possibly problematic conversion when passing
values to and from strbuf_fread)?
-- 8< --
Date: Sun, 2 Jan 2011 21:37:36 -0600
Subject: vcs-svn: make buffer_read_binary API more convenient
buffer_read_binary is a thin wrapper around fread, but its signature
is wrong:
- fread can fill an arbitrary in-memory buffer. buffer_read_binary
is limited to buffers whose size is representable by a 32-bit
integer.
- The result from fread is the number of bytes actually read.
buffer_read_binary only reports the number of bytes read by
incrementing sb->len by that amount and returns void.
Fix both: let buffer_read_binary accept a size_t instead of uint32_t
for the number of bytes to try to read and as a convenience return the
number of bytes read.
Signed-off-by: Jonathan Nieder <redacted>
---
vcs-svn/line_buffer.c | 6 +++---
vcs-svn/line_buffer.h | 2 +-
2 files changed, 4 insertions(+), 4 deletions(-)
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:50:55
Hi,
David Barr wrote:
As previously, I have tested against
the ASF subversion repository to increase confidence in the series.
Hopefully, this brings us a little closer to having full support for
version 3 of the subversion dump format in master.
Thanks much for digging this up. (To think, it's been half a year
since the svndiff0 parser was written!) I've queued everything except
patch 16, and grabbing patch 16 on top of db/delta-applier +
db/vcs-svn-incremental should be just a formality.
The patches have had over three months to cook already. Whether or
not a nice UI wrapper is ready[*], I'll be testing and then merging them
to svn-fe in the next couple of weeks. Sorry to take so long at this.
[*] which will probably mean finally figuring out Tomas's work
From: David Barr <hidden> Date: 2016-06-15 22:50:55
Hi,
Thanks much for digging this up. (To think, it's been half a year
since the svndiff0 parser was written!) I've queued everything except
patch 16, and grabbing patch 16 on top of db/delta-applier +
db/vcs-svn-incremental should be just a formality.
Once more, I'll run the tests just in case there's any regression.
(Having applied the aforementioned skipped patch.)
--
David Barr.
Hej,
I'm not sure if this was the origin email ...
Commit e7d04ee147dcbe6af1fa1d2147466696e is OK.
But:
failure on t9010 with commit 195b7ca6f229455da61f9f6b
=============
# test_cmp expect.message actual.message &&
# test_cmp expect.hello1 actual.hello1 &&
# test_cmp expect.hello2 actual.hello2
#
ok 14 - change file mode and reiterate content
ok 15 - deltas not supported
ok 16 - property deltas supported
ok 17 - properties on /
ok 18 - deltas for typechange
ok 19 - set up svn repo
ok 20 - t9135/svn.dump
# still have 3 known breakage(s)
# failed 1 among remaining 17 test(s)
1..20
=====================
Some more info:
b@birne:~/projects/git/git.git> uname -a
Darwin birne.lan 10.7.0 Darwin Kernel Version 10.7.0: Sat Jan 29
15:17:16 PST 2011; root:xnu-1504.9.37~1/RELEASE_I386 i386
tb@birne:~/projects/git/git.git> svn --version
svn, version 1.6.15 (r1038135)
compiled Jan 29 2011, 15:18:15
tb@birne:~/projects/git/git.git> svnadmin --version
svnadmin, version 1.6.15 (r1038135)
compiled Jan 29 2011, 15:18:15
which svn
/usr/bin/svn
I can assist with some more testing
BR
/Torsten
On 03/25/2011 05:07 AM, Jonathan Nieder wrote:
> A caller to buffer_read_string cannot easily tell the difference
> between the string "foo" followed by an early end of file and the
> string "foo\0bar\0baz". In a half-hearted attempt to catch early EOF,
> c9d1c8ba (2010-12-28) introduced a safety strlen(val) == len for
> property keys and values, to at least keep svn-fe from reading
> uninitialized data when a property list ends early due to EOF.
>
> But it is permissible for both keys and values to contain null
> characters, so in handling revision 59151 of the ASF repository svn-fe
> encounters a null byte and produces the following message:
>
> fatal: invalid dump: unexpected end of file
>
> Fix it by using buffer_read_binary to read to a strbuf (and keep track
> of the actual length read). Most consumers of properties still use
> C-style strings, so in practice we still can't use an author or log
> message with embedded nuls, but at least this way svn-fe won't error
> out.
>
> Reported-by: David Barr[off-list ref]
> Signed-off-by: Jonathan Nieder[off-list ref]
> ---
> t/t9010-svn-fe.sh | 27 +++++++++++++++++++++++++++
> vcs-svn/svndump.c | 24 ++++++++++--------------
> 2 files changed, 37 insertions(+), 14 deletions(-)
>
> diff --git a/t/t9010-svn-fe.sh b/t/t9010-svn-fe.sh
> index 5a6a4b9..47f1e4f 100755
> --- a/t/t9010-svn-fe.sh
> +++ b/t/t9010-svn-fe.sh
> @@ -370,6 +370,33 @@ test_expect_failure 'change file mode but keep
old content' '
> test_cmp hello actual.target
> '
>
> +test_expect_success 'null byte in property value' '
> + reinit_git&&
> + echo "commit message">expect.message&&
> + {
> + properties \
> + unimportant "something with a null byte (Q)" \
> + svn:log "commit message"&&
> + echo PROPS-END
> + } |
> + q_to_nul>props&&
> + {
> + cat<<-\EOF&&
> + SVN-fs-dump-format-version: 3
> +
> + Revision-number: 1
> + EOF
> + echo Prop-content-length: $(wc -c<props)&&
> + echo Content-length: $(wc -c<props)&&
> + echo&&
> + cat props
> + }>nullprop.dump&&
> + test-svn-fe nullprop.dump>stream&&
> + git fast-import<stream&&
> + git diff-tree --always -s --format=%s HEAD>actual.message&&
> + test_cmp expect.message actual.message
> +'
> +
> test_expect_success 'change file mode and reiterate content' '
> reinit_git&&
> cat>expect<<-\EOF&&
> diff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c
> index ea5b128..c00f031 100644
> --- a/vcs-svn/svndump.c
> +++ b/vcs-svn/svndump.c
> @@ -147,6 +147,7 @@ static void die_short_read(void)
> static void read_props(void)
> {
> static struct strbuf key = STRBUF_INIT;
> + static struct strbuf val = STRBUF_INIT;
> const char *t;
> /*
> * NEEDSWORK: to support simple mode changes like
> @@ -163,15 +164,15 @@ static void read_props(void)
> uint32_t type_set = 0;
> while ((t = buffer_read_line(&input))&& strcmp(t, "PROPS-END")) {
> uint32_t len;
> - const char *val;
> const char type = t[0];
> int ch;
>
> if (!type || t[1] != ' ')
> die("invalid property line: %s\n", t);
> len = atoi(&t[2]);
> - val = buffer_read_string(&input, len);
> - if (!val || strlen(val) != len)
> + strbuf_reset(&val);
> + buffer_read_binary(&input,&val, len);
> + if (val.len< len)
> die_short_read();
>
> /* Discard trailing newline. */
> @@ -179,22 +180,17 @@ static void read_props(void)
> if (ch == EOF)
> die_short_read();
> if (ch != '\n')
> - die("invalid dump: expected newline after %s", val);
> + die("invalid dump: expected newline after %s", val.buf);
>
> switch (type) {
> case 'K':
> + strbuf_swap(&key,&val);
> + continue;
> case 'D':
> - strbuf_reset(&key);
> - if (val)
> - strbuf_add(&key, val, len);
> - if (type == 'K')
> - continue;
> - assert(type == 'D');
> - val = NULL;
> - len = 0;
> - /* fall through */
> + handle_property(&val, NULL, 0,&type_set);
> + continue;
> case 'V':
> - handle_property(&key, val, len,&type_set);
> + handle_property(&key, val.buf, len,&type_set);
> strbuf_reset(&key);
> continue;
> default:
======================
On 03/25/2011 05:07 AM, Jonathan Nieder wrote:
quoted hunk
A caller to buffer_read_string cannot easily tell the difference
between the string "foo" followed by an early end of file and the
string "foo\0bar\0baz". In a half-hearted attempt to catch early EOF,
c9d1c8ba (2010-12-28) introduced a safety strlen(val) == len for
property keys and values, to at least keep svn-fe from reading
uninitialized data when a property list ends early due to EOF.
But it is permissible for both keys and values to contain null
characters, so in handling revision 59151 of the ASF repository svn-fe
encounters a null byte and produces the following message:
fatal: invalid dump: unexpected end of file
Fix it by using buffer_read_binary to read to a strbuf (and keep track
of the actual length read). Most consumers of properties still use
C-style strings, so in practice we still can't use an author or log
message with embedded nuls, but at least this way svn-fe won't error
out.
Reported-by: David Barr<redacted>
Signed-off-by: Jonathan Nieder<redacted>
---
t/t9010-svn-fe.sh | 27 +++++++++++++++++++++++++++
vcs-svn/svndump.c | 24 ++++++++++--------------
2 files changed, 37 insertions(+), 14 deletions(-)
@@ -370,6 +370,33 @@ test_expect_failure 'change file mode but keep old content' 'test_cmphelloactual.target'+test_expect_success'null byte in property value''+reinit_git&&+echo"commit message">expect.message&&+{+properties\+unimportant"something with a null byte (Q)"\+svn:log"commit message"&&+echoPROPS-END+}|+q_to_nul>props&&+{+cat<<-\EOF&&+SVN-fs-dump-format-version:3++Revision-number:1+EOF+echoProp-content-length:$(wc-c<props)&&+echoContent-length:$(wc-c<props)&&+echo&&+catprops+}>nullprop.dump&&+test-svn-fenullprop.dump>stream&&+gitfast-import<stream&&+gitdiff-tree--always-s--format=%sHEAD>actual.message&&+test_cmpexpect.messageactual.message+'+test_expect_success'change file mode and reiterate content''reinit_git&&cat>expect<<-\EOF&&
@@ -179,22 +180,17 @@ static void read_props(void)if(ch==EOF)die_short_read();if(ch!='\n')-die("invalid dump: expected newline after %s",val);+die("invalid dump: expected newline after %s",val.buf);switch(type){case'K':+strbuf_swap(&key,&val);+continue;case'D':-strbuf_reset(&key);-if(val)-strbuf_add(&key,val,len);-if(type=='K')-continue;-assert(type=='D');-val=NULL;-len=0;-/* fall through */+handle_property(&val,NULL,0,&type_set);+continue;case'V':-handle_property(&key,val,len,&type_set);+handle_property(&key,val.buf,len,&type_set);strbuf_reset(&key);continue;default:
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:50:55
tb wrote:
failure on t9010 with commit 195b7ca6f229455da61f9f6b
[...]
Darwin birne.lan 10.7.0 Darwin Kernel Version 10.7.0: Sat Jan 29
15:17:16 PST 2011; root:xnu-1504.9.37~1/RELEASE_I386 i386
Could you try 41e6b91f (vcs-svn: add missing cast to printf argument,
2011-03-27) from
git://repo.or.cz/git/jrn.git svn-fe
? I suspect this is fallout from a missing cast in the commit you
pointed to. Thanks for noticing.
failure on t9010 with commit 195b7ca6f229455da61f9f6b
[...]
quoted
Darwin birne.lan 10.7.0 Darwin Kernel Version 10.7.0: Sat Jan 29
15:17:16 PST 2011; root:xnu-1504.9.37~1/RELEASE_I386 i386
Could you try 41e6b91f (vcs-svn: add missing cast to printf argument,
2011-03-27) from
git://repo.or.cz/git/jrn.git svn-fe
? I suspect this is fallout from a missing cast in the commit you
pointed to. Thanks for noticing.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
No good news from my side:
Same (?) problem with 41e6b...
not ok - 13 NUL in log message, file content, and property name
/Torsten
Note:
(commit 41e6b works OK on my 32 bit 386 linux machine.
There is no svn on that machine, and commit 195b7ca6f22 had 10
failures in TC 9010. (If that is any useful information))
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:50:55
Torsten Bögershausen wrote:
No good news from my side:
Same (?) problem with 41e6b...
not ok - 13 NUL in log message, file content, and property name
Alas. Could you send the output for that test from
"sh t9010-svn-fe.sh -v -i", or from
GIT_TRACE=1 sh -x t9010-svn-fe.sh -v -i
if it looks more enlightening?