From: Jakub Narebski <hidden> Date: 2016-06-15 22:42:49
To make gitweb faster I thought about adding to it, or to Git.pm,
simple nonvalidation config file reader. Nonvalidating means that
it would accept some input which git-repo-config considers invalid.
Some of the trouble is caused because of coner cases like this
example
[section "sub ; # sect \" ion\\"] ; "]
key = a " b ; " ; " c ; "
which is valid, but strange.
I'm not proficent in Perl, so help is appreciated.
-- >8 --
#!/usr/bin/perl
use strict;
use warnings;
use Text::Balanced qw(extract_delimited);
sub read_config {
my $configfile = shift;
my $section = shift;
my %config;
open my $fd, $configfile
or die "Cannot open $configfile: $!";
my $sectfull;
while (my $line = <$fd>) {
chomp $line;
if ($line =~ m/^\s*\[\s*([^][:space:]]*)\s*\](.*)$/) {
# section without subsection
my $sect = lc($1);
$sectfull = $sect;
} elsif ($line =~ m/\s*\[([^][:space:]]*)\s"((?:\\.|[^"])*)"\](.*)$/) {
# section with subsection
my $sect = lc($1);
my $subsect = $2;
$subsect =~ s/\\(.)/$1/g;
$sectfull = "$sect.$subsect";
} elsif ($line =~ m/\s*(\w+)\s*=\s*(.*?)\s*$/) {
# variable assignment
my $key = lc($1);
my $rhs = $2;
my $value = '';
my ($next, $remainder, $prefix) = qw();
DELIM: {
do {
($next, $remainder, $prefix) =
extract_delimited($rhs, '"', qr/(?:\\.|[^"])*/);
if ($prefix =~ s/\s*[;#].*$//) {
# comment in unquoted part
$value .= $prefix;
last DELIM;
} else {
$value .= $prefix if $prefix;
if ($next && $next =~ s/^"(.*)"$/$1/) {
$value .= $next;
}
}
$rhs = $remainder;
} while ($rhs && $next);
} # DELIM:
if ($remainder) {
$remainder =~ s/\s*[;#].*$//;
$value .= $remainder;
}
$value =~ s/\\(.)/$1/g;
if (exists $config{"$sectfull.$key"}) {
push @{$config{"$sectfull.$key"}}, $value;
} else {
$config{"$sectfull.$key"} = [ $value ];
}
} elsif ($line =~ m/^\s*(\w+)\s*(:?[;#].*)?$/) {
# boolean variable without value
my $key = lc($1);
if (exists $config{"$sectfull.$key"}) {
push @{$config{"$sectfull.$key"}}, undef;
} else {
$config{"$sectfull.$key"} = [ undef ];
}
} # end if
}
close $fd
or die "Cannot close $configfile: $!";
return wantarray ? %config : \%config;
}
# --------------------------------------------------------------------------
my %config;
%config = read_config("~/git/.git/config");
%config = read_config("/tmp/jnareb/gitconfig");
foreach my $ckey (sort keys %config) {
foreach my $cvalue (@{$config{$ckey}}) {
if (defined $cvalue) {
print "$ckey=$cvalue\n";
} else {
print "$ckey\n";
}
}
}
__END__
From: Nikolai Weibull <hidden> Date: 2016-06-15 22:42:49
On 1/15/07, Johannes Schindelin [off-list ref] wrote:
On Mon, 15 Jan 2007, Nikolai Weibull wrote:
quoted
On 1/15/07, Johannes Schindelin [off-list ref] wrote:
quoted
quoted
On Mon, 15 Jan 2007, Eric Wong wrote:
quoted
quoted
quoted
quoted
Would you write "git repo-config --perl", then? ;-)
quoted
quoted
quoted
The below patch should be a start (only tested on my fairly standard
.git/config). A --python option should be easy, too :)
quoted
quoted
A bit shorter (and gets the booleans right, plus being even easier
towards --python extension):
quoted
If we're going down this slippery slope, why not just give up and add
a --xml switch instead?
AFAIR this switch was meant to _enhance_ performance.
As far as I can tell, comparing fork() vs. reading a dump with eval
vs. XML isn't meaningful - parsing a 20-line XML file can hardly be
much more (if it even is more) expensive than evaling a file of the
same length.
quoted
That said, parsing the config file as-is can't be so difficult that we
need to export it to separate files with a different syntax, now can it?
The point is having one parser to rule them all, and avoid having
different parsers, all with their own set of shortcomings.
So then you must agree that having one export format makes a lot of
sense, for the same reasons. Not that I think that an export format
makes sense in the first place.
nikolai
From: Eric Wong <hidden> Date: 2016-06-15 22:42:49
Jakub Narebski [off-list ref] wrote:
Eric Wong wrote:
quoted
Jakub Narebski [off-list ref] wrote:
quoted
To make gitweb faster I thought about adding to it, or to Git.pm,
simple nonvalidation config file reader. Nonvalidating means that
it would accept some input which git-repo-config considers invalid.
How about something like git-for-each-ref that dumps the entire output
of a config file into an eval()-able string? That way we don't have to
deal with corner-cases and subtle differences between C and Perl
implementations.
The idea is (at least for gitweb) to avoid cost of fork. And I think
if the format gets documented properly, there should be no differences
in config file parsing.
If the Perl output is redirected to a file (say .git/config.perl) and
only regenerated when .git/config changes, `do(".git/config.perl")' will
likely be faster since all the parsing will be done by Perl itself.
--
Eric Wong
From: Shawn O. Pearce <hidden> Date: 2016-06-15 22:42:49
Eric Wong [off-list ref] wrote:
If the Perl output is redirected to a file (say .git/config.perl) and
only regenerated when .git/config changes, `do(".git/config.perl")' will
likely be faster since all the parsing will be done by Perl itself.
So long as its automatic in gitweb.cgi and based on the stat
attributes of .git/config, OK. But my database background tells
me two copies of the same thing is fishy...
--
Shawn.
From: Jakub Narebski <hidden> Date: 2016-06-15 22:42:49
Eric Wong wrote:
Jakub Narebski [off-list ref] wrote:
quoted
Eric Wong wrote:
quoted
Jakub Narebski [off-list ref] wrote:
quoted
To make gitweb faster I thought about adding to it, or to Git.pm,
simple nonvalidation config file reader. Nonvalidating means that
it would accept some input which git-repo-config considers invalid.
How about something like git-for-each-ref that dumps the entire output
of a config file into an eval()-able string? That way we don't have to
deal with corner-cases and subtle differences between C and Perl
implementations.
The idea is (at least for gitweb) to avoid cost of fork. And I think
if the format gets documented properly, there should be no differences
in config file parsing.
If the Perl output is redirected to a file (say .git/config.perl) and
only regenerated when .git/config changes, `do(".git/config.perl")' will
likely be faster since all the parsing will be done by Perl itself.
Would you write "git repo-config --perl", then? ;-)
Besides, I'd rather avoid the need for /tmp/gitweb, and I think usually
gitweb do not have (and should not have) write access to repository.
--
Jakub Narebski
Poland
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:42:49
Hi,
On Mon, 15 Jan 2007, Eric Wong wrote:
quoted
Would you write "git repo-config --perl", then? ;-)
The below patch should be a start (only tested on my fairly standard
.git/config). A --python option should be easy, too :)
A bit shorter (and gets the booleans right, plus being even easier
towards --python extension):
---
builtin-repo-config.c | 19 +++++++++++++++++--
1 files changed, 17 insertions(+), 2 deletions(-)
From: Eric Wong <hidden> Date: 2016-06-15 22:42:49
Jakub Narebski [off-list ref] wrote:
Eric Wong wrote:
quoted
Jakub Narebski [off-list ref] wrote:
quoted
Eric Wong wrote:
quoted
Jakub Narebski [off-list ref] wrote:
quoted
To make gitweb faster I thought about adding to it, or to Git.pm,
simple nonvalidation config file reader. Nonvalidating means that
it would accept some input which git-repo-config considers invalid.
How about something like git-for-each-ref that dumps the entire output
of a config file into an eval()-able string? That way we don't have to
deal with corner-cases and subtle differences between C and Perl
implementations.
The idea is (at least for gitweb) to avoid cost of fork. And I think
if the format gets documented properly, there should be no differences
in config file parsing.
If the Perl output is redirected to a file (say .git/config.perl) and
only regenerated when .git/config changes, `do(".git/config.perl")' will
likely be faster since all the parsing will be done by Perl itself.
Would you write "git repo-config --perl", then? ;-)
The below patch should be a start (only tested on my fairly standard
.git/config). A --python option should be easy, too :)
Besides, I'd rather avoid the need for /tmp/gitweb, and I think usually
gitweb do not have (and should not have) write access to repository.
Good point. Having to maintain a .git/config.perl in the repository
would be a pain from an administrative standpoint; but on the other hand
.git/config is not often regenerated.
I don't think giving gitweb write access to a repo is a good idea;
either. Perhaps it would be updated via hook like the HTTP stuff.
IMHO, there is nothing wrong with gitweb writing to /tmp; however.
From: Nikolai Weibull <hidden> Date: 2016-06-15 22:42:49
On 1/15/07, Johannes Schindelin [off-list ref] wrote:
On Mon, 15 Jan 2007, Eric Wong wrote:
quoted
quoted
Would you write "git repo-config --perl", then? ;-)
quoted
The below patch should be a start (only tested on my fairly standard
.git/config). A --python option should be easy, too :)
A bit shorter (and gets the booleans right, plus being even easier
towards --python extension):
If we're going down this slippery slope, why not just give up and add
a --xml switch instead? Readable by all and a lot more flexible than
--perl, --python, --ruby, --tcl, --sh, --c++, --fortran, --lisp,
--html, --that-next-silver-bullet-language-that-hasnt-been-invented-yet-but-will-need-its-own-switch-once-it-has-been.
That said, parsing the config file as-is can't be so difficult that we
need to export it to separate files with a different syntax, now can
it?
nikolai
From: Jakub Narebski <hidden> Date: 2016-06-15 22:42:49
Nikolai Weibull wrote:
On 1/15/07, Johannes Schindelin [off-list ref] wrote:
quoted
On Mon, 15 Jan 2007, Eric Wong wrote:
quoted
quoted
quoted
Would you write "git repo-config --perl", then? ;-)
quoted
quoted
The below patch should be a start (only tested on my fairly standard
.git/config). A --python option should be easy, too :)
quoted
A bit shorter (and gets the booleans right, plus being even easier
towards --python extension):
If we're going down this slippery slope, why not just give up and add
a --xml switch instead? Readable by all and a lot more flexible than
--perl, --python, --ruby, --tcl, --sh, --c++, --fortran, --lisp,
--html, --that-next-silver-bullet-language [...].
That said, parsing the config file as-is can't be so difficult that we
need to export it to separate files with a different syntax, now can
it?
Parsing the config file is not _that_ difficult (the first post in this
thread had config reader in Perl), but it is not that easy: case
(in)setiviness, quoting, escaping, comments, removing leading and
trailing whitespace when not quoted...
P.S. I'd rather have an additional implementation (in Perl) conforming
to yet to be written git ini-like config file specs, to find places
where canonic parser, git-repo-config, doesn't conform to the specs.
--
Jakub Narebski
Poland
From: Jakub Narebski <hidden> Date: 2016-06-15 22:42:49
Eric Wong wrote:
Jakub Narebski [off-list ref] wrote:
quoted
To make gitweb faster I thought about adding to it, or to Git.pm,
simple nonvalidation config file reader. Nonvalidating means that
it would accept some input which git-repo-config considers invalid.
How about something like git-for-each-ref that dumps the entire output
of a config file into an eval()-able string? That way we don't have to
deal with corner-cases and subtle differences between C and Perl
implementations.
The idea is (at least for gitweb) to avoid cost of fork. And I think
if the format gets documented properly, there should be no differences
in config file parsing.
Please remember also that is first draft of git config file reader
in perl; an alpha version.
--
Jakub Narebski
Poland
From: Eric Wong <hidden> Date: 2016-06-15 22:42:49
Jakub Narebski [off-list ref] wrote:
To make gitweb faster I thought about adding to it, or to Git.pm,
simple nonvalidation config file reader. Nonvalidating means that
it would accept some input which git-repo-config considers invalid.
How about something like git-for-each-ref that dumps the entire output
of a config file into an eval()-able string? That way we don't have to
deal with corner-cases and subtle differences between C and Perl
implementations.
--
Eric Wong
From: Eric Wong <hidden> Date: 2016-06-15 22:42:49
Johannes Schindelin [off-list ref] wrote:
Hi,
On Mon, 15 Jan 2007, Eric Wong wrote:
quoted
quoted
Would you write "git repo-config --perl", then? ;-)
The below patch should be a start (only tested on my fairly standard
.git/config). A --python option should be easy, too :)
A bit shorter (and gets the booleans right, plus being even easier
towards --python extension):
Your version doesn't get arrays right, however.
Here's a Perl/Python/Ruby version below. It should be extendable for
other languages; feedback and additions appreciated:
Note that usage has been changed to --dump=(perl|python|ruby)
I may add key_suffix to lang_dump just to be consistent with pairings,
but array_start seems to handle all cases of it and it would be
redundant...
@@ -14,6 +15,90 @@ static int do_not_match;staticintseen;staticenum{T_RAW,T_INT,T_BOOL}type=T_RAW;+structlang_dump{+constchar*name;+constchar*decl_start;+constchar*decl_end;+constchar*key_prefix;+constchar*array_start;+constchar*array_end;+constchar*val_prefix;+constchar*val_suffix;+constchar*true_val;/* should already be quoted, if needed */+void(*quote_key_fn)(FILE*,constchar*);+void(*quote_val_fn)(FILE*,constchar*);+};+staticchar*last_key;+staticstructlang_dump*lang;+staticstructlang_dumplang_dump_defs[]={+{"perl",+"\%git_config = (\n",");\n",+"\t",+" => [\n","\t],\n",+"\t\t",",\n",+"'true'",+perl_quote_print,perl_quote_print},+{"python",+"git_config = {\n","}\n",+" ",+" : [\n"," ],\n",+" ",",\n",+"True",+python_quote_print,python_quote_print},+{"ruby",/* Ruby is very Perl-like */+"git_config = {\n","}\n",+" ",+" => [\n"," ],\n",+" ",",\n",+"true",+perl_quote_print,perl_quote_print},+};++staticintshow_lang_config(constchar*key_,constchar*value_)+{+if(last_key){+if(strcmp(last_key,key_)){+free(last_key);+fputs(lang->array_end,stdout);+gotonew_key;+}+}else{+new_key:+last_key=xstrdup(key_);+fputs(lang->key_prefix,stdout);+lang->quote_key_fn(stdout,key_);+fputs(lang->array_start,stdout);+}+fputs(lang->val_prefix,stdout);+if(value_)+lang->quote_val_fn(stdout,value_);+else+fputs(lang->true_val,stdout);+fputs(lang->val_suffix,stdout);+return0;+}++staticintshow_lang_config_all(constchar*lang_name)+{+inti,rv;+for(i=ARRAY_SIZE(lang_dump_defs);--i>=0;){+if(strcmp(lang_name,lang_dump_defs[i].name))+continue;+lang=lang_dump_defs+i;+fputs(lang->decl_start,stdout);+rv=git_config(show_lang_config);+if(last_key){+free(last_key);+last_key=NULL;+fputs(lang->array_end,stdout);+fputs(lang->decl_end,stdout);+}+returnrv;+}+fputs("Dumping config to '%s' is not yet supported",stderr);+return-1;+}+staticintshow_all_config(constchar*key_,constchar*value_){if(value_)
@@ -14,6 +15,90 @@ static int do_not_match;staticintseen;staticenum{T_RAW,T_INT,T_BOOL}type=T_RAW;+structlang_dump{+constchar*name;+constchar*decl_start;+constchar*decl_end;+constchar*key_prefix;+constchar*array_start;+constchar*array_end;+constchar*val_prefix;+constchar*val_suffix;+constchar*true_val;/* should already be quoted, if needed */+void(*quote_key_fn)(FILE*,constchar*);+void(*quote_val_fn)(FILE*,constchar*);+};+staticchar*last_key;+staticstructlang_dump*lang;+staticstructlang_dumplang_dump_defs[]={+{"perl",+"\%git_config = (\n",");\n",
So this makes _all_ config vars arrays? It is consistent, yes... but it is
also ugly, no?
+static int show_lang_config_all(const char *lang_name)
+{
+ int i, rv;
+ for (i = ARRAY_SIZE(lang_dump_defs); --i >= 0; ) {
+ if (strcmp(lang_name, lang_dump_defs[i].name))
+ continue;
+ lang = lang_dump_defs + i;
IMHO this would be much easier to read using a path_list:
struct path_list_item *item = path_list_lookup(lang_name, &langs);
if (item == NULL)
return -1;
lang = item->util;
The two quote members seem to be the same for _all_ three languages.
Yes, I was trying to imagine a corner case where quoting
for keys could be different than values.
I know Ruby can use :symbols but those don't support '.' and '-' as
far as I know, Perl doesn't require quoting for keys if they match
^-?\w+. I guess just using quoted strings as keys is fine enough.
The patch below uses the same quote operator for both keys and values.
quoted
+ { "python",
+ "git_config = {\n", "}\n",
+ " ",
I don't understand why you do not consolidate that into using tabs for
_all_ backends?
I wanted to make things look familiar to people using those languages.
Most Ruby programmers I've seen use 2-space indents. I myself have given
into using them when I write Ruby.
Python doesn't seem to care about indentation for data structures, but I
think Python programmers prefer spaces for indentation. I'm not very
experienced with Python, however.
Tabs are my own personal preference for Perl; but indentation is very
inconsistent in Perl code I've looked at :/. perlstyle(1) actually
recommends 4 space indents...
On the other hand, we are writing for interpreters and not humans. So
maybe just using tabs is good enough (some formatting makes debugging
easier, so I'm not putting everything on one line :).
So this makes _all_ config vars arrays? It is consistent, yes... but it is
also ugly, no?
Somewhat ugly, yes, but I think returning everything as an array would
make things easier for code using the data structures. They could
always just reference the first element if they didn't want the array
instead of having to find the type with ref() or .kind_of?
quoted
+static int show_lang_config_all(const char *lang_name)
+{
+ int i, rv;
+ for (i = ARRAY_SIZE(lang_dump_defs); --i >= 0; ) {
+ if (strcmp(lang_name, lang_dump_defs[i].name))
+ continue;
+ lang = lang_dump_defs + i;
IMHO this would be much easier to read using a path_list:
struct path_list_item *item = path_list_lookup(lang_name, &langs);
if (item == NULL)
return -1;
lang = item->util;
Ah, I didn't know about path_list_lookup(). Now that I know about it, I
don't think it's worth it to create the extra data structures around
it. We can just as easily switch to bsearch(3) when we add more
languages.
@@ -14,6 +15,89 @@ static int do_not_match;staticintseen;staticenum{T_RAW,T_INT,T_BOOL}type=T_RAW;+structlang_dump{+constchar*name;+constchar*decl_start;+constchar*decl_end;+constchar*key_prefix;+constchar*array_start;+constchar*array_end;+constchar*val_prefix;+constchar*val_suffix;+constchar*true_val;/* should already be quoted, if needed */+void(*quote_fn)(FILE*,constchar*);+};+staticchar*last_key;+staticstructlang_dump*lang;+staticstructlang_dumplang_dump_defs[]={+{"perl",+"\%git_config = (\n",");\n",+"\t",+" => [\n","\t],\n",+"\t\t",",\n",+"'true'",+perl_quote_print},+{"python",+"git_config = {\n","}\n",+" ",+" : [\n"," ],\n",+" ",",\n",+"True",+python_quote_print},+{"ruby",/* Ruby is very Perl-like */+"git_config = {\n","}\n",+" ",+" => [\n"," ],\n",+" ",",\n",+"true",+perl_quote_print},+};++staticintshow_lang_config(constchar*key_,constchar*value_)+{+if(last_key){+if(strcmp(last_key,key_)){+free(last_key);+fputs(lang->array_end,stdout);+gotonew_key;+}+}else{+new_key:+last_key=xstrdup(key_);+fputs(lang->key_prefix,stdout);+lang->quote_fn(stdout,key_);+fputs(lang->array_start,stdout);+}+fputs(lang->val_prefix,stdout);+if(value_)+lang->quote_fn(stdout,value_);+else+fputs(lang->true_val,stdout);+fputs(lang->val_suffix,stdout);+return0;+}++staticintshow_lang_config_all(constchar*lang_name)+{+inti,rv;+for(i=ARRAY_SIZE(lang_dump_defs);--i>=0;){+if(strcmp(lang_name,lang_dump_defs[i].name))+continue;+lang=lang_dump_defs+i;+fputs(lang->decl_start,stdout);+rv=git_config(show_lang_config);+if(last_key){+free(last_key);+last_key=NULL;+fputs(lang->array_end,stdout);+}+fputs(lang->decl_end,stdout);+returnrv;+}+fputs("Dumping config to '%s' is not yet supported",stderr);+return-1;+}+staticintshow_all_config(constchar*key_,constchar*value_){if(value_)