From: Junio C Hamano <hidden> Date: 2016-06-15 22:50:26
Ramkumar Ramachandra [off-list ref] writes:
Introduce a new command-line option --inline-blobs that always inlines
blobs instead of referring to them via marks or their original SHA-1
hash.
Signed-off-by: Ramkumar Ramachandra <redacted>
---
Documentation/git-fast-export.txt | 5 +++++
builtin/fast-export.c | 23 +++++++++++++++++++++--
2 files changed, 26 insertions(+), 2 deletions(-)
Hmm, this smells somewhat fishy.
Wasn't G-F-I designed to be a common stream format for other SCMs to
generate streams, so that importers and exporters can be written once for
each SCM to interoperate?
This patch will allow you to write an importer that can only take a stream
with inlined blobs without any references to previous occurrences, but if
the exporter for an SCM that you are trying to interoperate with does not
support --inline-blobs, you are screwed.
What is the problem you are really trying to solve? If it is "it is
cumbersome to keep track of blob references", wouldn't it be nicer to
instead make it easier for importers to support referenced blobs?
Just thinking aloud, but is it possible to write a filter that converts an
arbitrary G-F-I stream with referenced blobs into a G-F-I stream without
referenced blobs by inlining all the blobs? Then other people do not have
to implement --inline-blobs to their own exporter.
If that is not possible, should/can we do something to at least allow
people to check if an existing stream is compatible with an importer that
cannot take referenced blobs without actually trying to run import (and
see it fail)? Do we need a way to encourage people to add --inline-blobs
support to their exporters? I suspect this series leads to make G-F-I
less useful by fragmenting the compatible subset of stream formats without
such effort, no?
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:50:26
Junio C Hamano wrote:
Ramkumar Ramachandra [off-list ref] writes:
quoted
Introduce a new command-line option --inline-blobs that always inlines
blobs instead of referring to them via marks or their original SHA-1
hash.
[...]
Hmm, this smells somewhat fishy.
Wasn't G-F-I designed to be a common stream format for other SCMs to
generate streams, so that importers and exporters can be written once for
each SCM to interoperate?
Here is one way to sell it:
With the inline blobs feature, fast-import backends have to
maintain less state. Using it should speed up exporting.
This is made optional because ...
I haven't thought through whether it ought to be optional or measured
the effect on import performance.
A separate question is what an svn fast-import backend should do with
all those blobs that are not ready to be written to dump. As a hack
while prototyping, one can rely on the "current" fast-export output,
even though that is not flexible or futureproof. Longer term, the
folllowing sounds very interesting
Just thinking aloud, but is it possible to write a filter that converts an
arbitrary G-F-I stream with referenced blobs into a G-F-I stream without
referenced blobs by inlining all the blobs?
to avoid complexity in the svn fast-import backend itself.
(Complicating detail: such a filter would presumably take responsibility
for --export-marks, so it might want a way to retrieve commit marks
from its downstream.)
Introduce a new command-line option --inline-blobs that always inlines
blobs instead of referring to them via marks or their original SHA-1
hash.
[...]
quoted
Hmm, this smells somewhat fishy.
Wasn't G-F-I designed to be a common stream format for other SCMs to
generate streams, so that importers and exporters can be written once for
each SCM to interoperate?
Here is one way to sell it:
With the inline blobs feature, fast-import backends have to
maintain less state. Using it should speed up exporting.
This is made optional because ...
I haven't thought through whether it ought to be optional or measured
the effect on import performance.
It simplifies other fast-import backends greatly, because persisting
blobs can be complicated and expensive. I was thinking of making
svn-fe support both inlined blobs, and blobs referenced by marks. When
it's possible to be cheap by optionally having inlined blobs, why not
optionally have them? The filter we develop later can be used for
older fast-import streams that don't have inlined blobs.
On a related note, does it make sense to version our fast-import
stream format? It's certainly going to keep evolving with time, and we
need backward compatibility.
A separate question is what an svn fast-import backend should do with
all those blobs that are not ready to be written to dump. As a hack
while prototyping, one can rely on the "current" fast-export output,
even though that is not flexible or futureproof. Longer term, the
folllowing sounds very interesting
Good point. The functionality to persist blobs that are refenced by
marks probably shouldn't be in svn-fe at all.
quoted
Just thinking aloud, but is it possible to write a filter that converts an
arbitrary G-F-I stream with referenced blobs into a G-F-I stream without
referenced blobs by inlining all the blobs?
to avoid complexity in the svn fast-import backend itself.
(Complicating detail: such a filter would presumably take responsibility
for --export-marks, so it might want a way to retrieve commit marks
from its downstream.)
This filter will need to persist every blob for the entire lifetime of
the program. We can't possibly do it in-memory, so we have to find
some way to persist them on-disk and retrieve them very
quickly. Jonathan suggested using something like ToyoCabinet earlier-
I'll start working and see what I come up with.
-- Ram
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:50:26
Junio C Hamano wrote:
Just thinking aloud, but is it possible to write a filter that converts an
arbitrary G-F-I stream with referenced blobs into a G-F-I stream without
referenced blobs by inlining all the blobs?
A few details to watch out for:
- A mark, as in
M 100644 :1 path/to/file
can refer to a blob from a previous import. A mark can even refer
to a manually prepared marks file.
- The syntax
M 100644 0409ac9fd3f1ea36680189e07116e58b2630ccad path/to/file
refers to a blob that might not have been mentioned elsewhere in the
stream. This is the variant used by "git fast-export --no-data" to
avoid transferring blob data. (In general, non-git backends would
presumably use something other than git blob IDs if they use this
feature. A filter of the kind we are describing would probably pass
them through.)
These datarefs can be acquired out of band (probably not a big deal)
or by using the "ls" command to copy from a previous revision:
> ls :3 "path/to/other/file"
100644 blob 0409ac9fd3f1ea36680189e07116e58b2630ccad git.c
> M 100644 0409ac9fd3f1ea36680189e07116e58b2630ccad path/to/file
- The cat-blob command ("cat-blob :1") allows frontends to request
the content of a previously imported blob (presumably in order to
apply a delta to it).
So while something like the filter you describe seems possible, it
cannot be as simple as
mkfifo replies &&
fast-export-frontend 3<replies |
inline-blobs |
fast-import-backend --cat-blob-fd=3 <args> 3>replies
for general <frontend> and <backend>. The frontend might try to cat
blobs by mark number or to pick off where it left off in a previous
run using a marks file.
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:50:26
Ramkumar Ramachandra wrote:
The functionality to persist blobs that are refenced by
marks probably shouldn't be in svn-fe at all.
Do you mean svn-fi?
This filter will need to persist every blob for the entire lifetime of
the program.
Depending on the interface, couldn't it be possible to rely on svn for
the content of blobs that have already been exported? If so, one
would only need a place to stash (1) a mapping from mark numbers to
(svn rev, path) pairs and (2) the full text of blobs that have not
been exported as part of a rev yet.
Cheers,
Jonathan
The functionality to persist blobs that are refenced by
marks probably shouldn't be in svn-fe at all.
Do you mean svn-fi?
Yeah. Sorry about the typo. Just to make it clear:
svn-fi should only ever support inlined blobs. For older streams,
there's an extra overhead- it'll have to be chained along with a
helper program that transforms the stream to inline all the blobs.
quoted
This filter will need to persist every blob for the entire lifetime of
the program.
Depending on the interface, couldn't it be possible to rely on svn for
the content of blobs that have already been exported? If so, one
would only need a place to stash (1) a mapping from mark numbers to
(svn rev, path) pairs and (2) the full text of blobs that have not
been exported as part of a rev yet.
Oh yes. We discussed this on IRC :) I'm just afraid that it won't be
fast enough- my idea is to essentially use `svnrdump dump` to replay
the blobs in a certain (path, revision); let me know if you think
there's a quicker way.
-- Ram
On Thu, 2011-01-20 at 10:20 +0530, Ramkumar Ramachandra wrote:
Hi,
Jonathan Nieder writes:
quoted
Junio C Hamano wrote:
quoted
Ramkumar Ramachandra [off-list ref] writes:
Just thinking aloud, but is it possible to write a filter that converts an
arbitrary G-F-I stream with referenced blobs into a G-F-I stream without
referenced blobs by inlining all the blobs?
to avoid complexity in the svn fast-import backend itself.
(Complicating detail: such a filter would presumably take responsibility
for --export-marks, so it might want a way to retrieve commit marks
from its downstream.)
This filter will need to persist every blob for the entire lifetime of
the program. We can't possibly do it in-memory, so we have to find
some way to persist them on-disk and retrieve them very
quickly. Jonathan suggested using something like ToyoCabinet earlier-
I'll start working and see what I come up with.
Is it worth including the extra dependency? Most systems that I'm in
frequent contact with already have some lightweight BDB implementation
already. I don't currently know of any with TokyoCabinet (or
KyotoCabinet for that matter) already in place. Besides, if all you're
doing is persisting blobs that you're likely to write out to disk
eventually anyway you might as well just do so once you have them and
keep an "index" (not to be confused with the Git Index, just lacking a
better word right now) of what you have in some standard in-memory
format (a heap?). From there you can build each commit into the Git
Index in the proper order once you have the required parts for
each--perhaps even re-using the blobs you've already dumped to disk
(mv'ing them or something).
Granted, there's a good likelihood that I'm missing something here, but
I don't see the point of adding external complexity (beyond what you're
currently stuck dealing with).
--
-Drew Northup
________________________________________________
"As opposed to vegetable or mineral error?"
-John Pescatore, SANS NewsBites Vol. 12 Num. 59
On Thu, 2011-01-20 at 10:20 +0530, Ramkumar Ramachandra wrote:
quoted
Hi,
Jonathan Nieder writes:
quoted
Junio C Hamano wrote:
quoted
Ramkumar Ramachandra [off-list ref] writes:
Just thinking aloud, but is it possible to write a filter that converts an
arbitrary G-F-I stream with referenced blobs into a G-F-I stream without
referenced blobs by inlining all the blobs?
to avoid complexity in the svn fast-import backend itself.
(Complicating detail: such a filter would presumably take responsibility
for --export-marks, so it might want a way to retrieve commit marks
from its downstream.)
This filter will need to persist every blob for the entire lifetime of
the program. We can't possibly do it in-memory, so we have to find
some way to persist them on-disk and retrieve them very
quickly. Jonathan suggested using something like ToyoCabinet earlier-
I'll start working and see what I come up with.
Is it worth including the extra dependency? Most systems that I'm in
frequent contact with already have some lightweight BDB implementation
already. I don't currently know of any with TokyoCabinet (or
KyotoCabinet for that matter) already in place. Besides, if all you're
doing is persisting blobs that you're likely to write out to disk
eventually anyway you might as well just do so once you have them and
keep an "index" (not to be confused with the Git Index, just lacking a
better word right now) of what you have in some standard in-memory
format (a heap?). From there you can build each commit into the Git
Index in the proper order once you have the required parts for
each--perhaps even re-using the blobs you've already dumped to disk
(mv'ing them or something).
Agreed. I wouldn't like to introduce an extra dependency either. I was
talking about using it for prototyping- if the final version includes
an extra dependency, it's unlikely to get merged into git.git :) The
final design will probably use an in-memory B+ tree, but I haven't
thought about that hard enough.
-- Ram
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:50:27
Ramkumar Ramachandra wrote:
Agreed. I wouldn't like to introduce an extra dependency either. I was
talking about using it for prototyping- if the final version includes
an extra dependency, it's unlikely to get merged into git.git :) The
final design will probably use an in-memory B+ tree, but I haven't
thought about that hard enough.
Immediate reaction:
Please no. There is a value to simplicity.
As you mention, the final form is a way off, so as long as people are
careful not to get locked into bad implementation decisions, I think
it is okay. I have refrained from nitpicking the implementation so
far because the design and interface are not obvious yet.
In this particular case, I am dreaming that we will discover a hidden
"mkdir -p" node-action in the dumpfile format so the list of
directories will not be needed. ;-)
Re Junio's critique:
is it possible to use
1) a table with callbacks? See the source code to unifdef for
inspiration.
2) separate code paths for different input states? fast-import.c
does this.
3) separate "parsing" and "acting" code? That can open the door to a
little paralellism, though not necessarily enough to matter.
parser actor
read a chunk
determine the first "thing to do"
pick up the first "thing to do"
do it
determine the next "thing to do"
pick up the next "thing to do"
do it
There is potential parallelism because the parser can keep
chugging along if the actor is blocked, say, writing its
output to the network.
Syntactically: the actor function (write_dump) calls the parser
function (next_command) to ask what to do next. If wanted, a later
refactoring could make that parser function just grab an action off
of a queue, while the parser proper runs in the background.
And of course if the "thing to do" data structure is simple enough,
this can also make the code easier to read.
I mention these ideas because some of them (especially #2) could make
the prototyping a lot easier as well as resulting in code that is
easier to review.
Re error handling:
Writing robust code (e.g., checking for errors) is also a lot easier
when done from the start. The svn-fe error handling is known to be a
problem (see BUGS in contrib/svn-fe/svn-fe.txt). So yes, I also
consider avoiding segfaults and deadlocks and catching parse errors to
be worthwhile things.
Thanks.
Jonathan