From: Michael Haggerty <hidden> Date: 2016-06-15 23:04:11
Sorry for the long email. Feel free to skip over the background info
and continue reading at "My solution" below.
This odyssey started with a typo:
$ git shortlog -snl v2.2.0..v2.3.0
14795 Junio C Hamano
1658 Jeff King
1400 Shawn O. Pearce
1109 Linus Torvalds
760 Jonathan Nieder
[...]
It took me a minute to realize why so many commits are listed. It
turns out that "-l", which I added by accident, requires an integer
argument, but it happily swallows "v2.2.0..v2.3.0" without error. This
leaves no range argument, so the command traverses the entire history
of HEAD.
The relevant code is
else if ((argcount = short_opt('l', av, &optarg))) {
options->rename_limit = strtoul(optarg, NULL, 10);
return argcount;
}
, which is broken in many ways. First of all, strtoul() is way too
permissive for simple tasks like this:
* It allows leading whitespace.
* It allows arbitrary trailing characters.
* It allows a leading sign character ('+' or '-'), even though the
result is unsigned.
* If the string doesn't contain a number at all, it sets its "endptr"
argument to point at the start of the string but doesn't set errno.
* If the value is larger than fits in an unsigned long, it returns the
value clamped to the range 0..ULONG_MAX (setting errno to ERANGE).
* If the value is between -ULONG_MAX and 0, it returns the positive
integer with the same bit pattern, without setting errno(!) (I can
hardly think of a scenario where this would be useful.)
* For base=0 (autodetect base), it allows a base specifier prefix "0x"
or "0" and parses the number accordingly. For base=16 it also allows
a "0x" prefix.
strtol() has similar foibles.
The caller compounds the permissivity of strtoul() with further sins:
* It does absolutely no error detection.
* It assigns the return value, which is an unsigned long, to an int
value. This could truncate the result, perhaps even resulting in
rename_limit being set to a negative value.
When I looked around, I found scores of sites that call atoi(),
strtoul(), and strtol() carelessly. And it's understandable. Calling
any of these functions safely is so much work as to be completely
impractical in day-to-day code.
git-compat-util.h has two functions, strtoul_ui() and strtol_i(), that
try to make parsing integers a little bit easier. But they are only
used in a few places, they hardly restrict the pathological
flexibility of strtoul()/strtol(), strtoul_ui() doesn't implement
clamping consistently when converting from unsigned long to unsigned
int, and neither function can be used when the integer value *is*
followed by other characters.
My solution: the numparse module
So I hereby propose a new module, numparse, to make it easier to parse
integral values from strings in a convenient, safe, and flexible way.
The first commit introduces the new module, and subsequent commits
change a sample (a small fraction!) of call sites to use the new
functions. Consider it a proof-of-concept. If people are OK with this
approach, I will continue sending patches to fix other call sites. (I
already have another two dozen such patches in my repo).
Please see the docstrings in numparse.h in the first commit for
detailed API docs. Briefly, there are eight new functions:
parse_{l,ul,i,ui}(const char *s, unsigned int flags,
T *result, char **endptr);
convert_{l,ul,i,ui}(const char *s, unsigned int flags, T *result);
The parse_*() functions are for parsing a number from the start of a
string; the convert_*() functions are for parsing a string that
consists of a single number. The flags argument selects not only the
base of the number, but also which of strtol()/strtoul()'s many
features should be allowed. The *_i() and *.ui() functions are for
parsing int and unsigned int values; they are careful about how they
truncate the corresponding long values. The functions all return 0 on
success and a negative number on error.
Here are a few examples of how these functions can be used (taken from
the header of numparse.h):
* Convert hexadecimal string s into an unsigned int. Die if there are
any characters in s besides hexadecimal digits, or if the result
exceeds the range of an unsigned int:
if (convert_ui(s, 16, &result))
die("...");
* Read a base-ten long number from the front of a string, allowing
sign characters and setting endptr to point at any trailing
characters:
if (parse_l(s, 10 | NUM_SIGN | NUM_TRAILING, &result, &endptr))
die("...");
* Convert decimal string s into a signed int, but not allowing the
string to contain a '+' or '-' prefix (and thereby indirectly
ensuring that the result will be non-negative):
if (convert_i(s, 10, &result))
die("...");
* Convert s into a signed int, interpreting prefix "0x" to mean
hexadecimal and "0" to mean octal. If the value doesn't fit in an
unsigned int, set result to INT_MIN or INT_MAX.
if (convert_i(s, NUM_SLOPPY, &result))
die("...");
My main questions:
* Do people like the API? My main goal was to make these functions as
painless as possible to use correctly, because there are so many
call sites.
* Is it too gimmicky to encode the base together with other options in
`flags`? (I think it is worth it to avoid the need for another
parameter, which callers could easily put in the wrong order.)
* Am I making callers too strict? In many cases where a positive
integer is expected (e.g., "--abbrev=<num>"), I have been replacing
code like
result = strtoul(s, NULL, 10);
with
if (convert_i(s, 10, &result))
die("...");
Strictly speaking, this is backwards incompatible, because the
former allows things like
--abbrev="+10"
--abbrev=" 10"
--abbrev="10 bananas"
--abbrev=-18446744073709551606
(all equivalent to "--abbrev=10"), whereas the new code rejects all
of them. It would be easy to change the new code to allow the first
three, by adding flags NUM_PLUS, NUM_LEADING_WHITESPACE, or
NUM_TRAILING respectively. But my inclination is to be strict
(though I could easily be persuaded to permit the first one).
* Should I submit tiny patches, each rewriting a single call site, or
make changes in larger chunks (say, file by file)? For example, in
revision.c there is a function handle_revision_opt() that contains
about 10 call sites that should be rewritten, for 10 different
options that take integers. My gut feeling is that the rewrite of
"--min-age=<value>" should be done in a different patch than that of
"--min-parents=<value>", especially since the invocations might use
different levels of parsing strictness that might be topics of
discussion on the ML. On the other hand, I feel silly bombarding the
list with tons of tiny patches.
* I couldn't think of any places where today's sloppy parsing could
result in security vulnerabilities, but people should think about
this, too. I would be especially wary of sites that call strtoul()
and assign the result to an int, seemingly not considering that the
result could end up negative.
These patches apply to "master". They are also available on my GitHub
repo [1].
Michael
[1] https://github.com/mhagger/git.git, branch "numparse1"
Michael Haggerty (14):
numparse: new module for parsing integral numbers
cacheinfo_callback(): use convert_ui() when handling "--cacheinfo"
write_subdirectory(): use convert_ui() for parsing mode
handle_revision_opt(): use skip_prefix() in many places
handle_revision_opt(): use convert_i() when handling "-<digit>"
strtoul_ui(), strtol_i(): remove functions
handle_revision_opt(): use convert_ui() when handling "--abbrev="
builtin_diff(): detect errors when parsing --unified argument
opt_arg(): val is always non-NULL
opt_arg(): use convert_i() in implementation
opt_arg(): report errors parsing option values
opt_arg(): simplify pointer handling
diff_opt_parse(): use convert_i() when handling "-l<num>"
diff_opt_parse(): use convert_i() when handling --abbrev=<num>
Makefile | 1 +
builtin/update-index.c | 3 +-
contrib/convert-objects/convert-objects.c | 3 +-
diff.c | 55 ++++----
git-compat-util.h | 26 ----
numparse.c | 180 ++++++++++++++++++++++++++
numparse.h | 207 ++++++++++++++++++++++++++++++
revision.c | 64 ++++-----
8 files changed, 447 insertions(+), 92 deletions(-)
create mode 100644 numparse.c
create mode 100644 numparse.h
--
2.1.4
From: Michael Haggerty <hidden> Date: 2016-06-15 23:04:11
Use convert_ui() instead of strtoul_ui() to parse the <mode> argument.
This tightens up the parsing a bit:
* Leading whitespace is no longer allowed
* '+' and '-' are no longer allowed
Signed-off-by: Michael Haggerty <redacted>
---
builtin/update-index.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
From: Michael Haggerty <hidden> Date: 2016-06-15 23:04:11
This reduces the need for "magic numbers".
Signed-off-by: Michael Haggerty <redacted>
---
revision.c | 59 ++++++++++++++++++++++++++++++-----------------------------
1 file changed, 30 insertions(+), 29 deletions(-)
From: Michael Haggerty <hidden> Date: 2016-06-15 23:04:11
This tightens up the parsing a bit:
* Leading whitespace is no longer allowed
* '+' and '-' are no longer allowed
It also removes the need to check separately that max_count is
non-negative.
Signed-off-by: Michael Haggerty <redacted>
---
revision.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
@@ -1709,8 +1710,7 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **argreturnargcount;}elseif((*arg=='-')&&isdigit(arg[1])){/* accept -<digit>, like traditional "head" */-if(strtol_i(arg+1,10,&revs->max_count)<0||-revs->max_count<0)+if(convert_i(arg+1,10,&revs->max_count))die("'%s': not a non-negative integer",arg+1);revs->no_walk=0;}elseif(!strcmp(arg,"-n")){
@@ -0,0 +1,180 @@+#include"git-compat-util.h"+#include"numparse.h"++#define NUM_NEGATIVE (1 << 16)+++staticintparse_precheck(constchar*s,unsignedint*flags)+{+constchar*number;++if(isspace(*s)){+if(!(*flags&NUM_LEADING_WHITESPACE))+return-NUM_LEADING_WHITESPACE;+do{+s++;+}while(isspace(*s));+}++if(*s=='+'){+if(!(*flags&NUM_PLUS))+return-NUM_PLUS;+number=s+1;+*flags&=~NUM_NEGATIVE;+}elseif(*s=='-'){+if(!(*flags&NUM_MINUS))+return-NUM_MINUS;+number=s+1;+*flags|=NUM_NEGATIVE;+}else{+number=s;+*flags&=~NUM_NEGATIVE;+}++if(!(*flags&NUM_BASE_SPECIFIER)){+intbase=*flags&NUM_BASE_MASK;+if(base==0){+/* This is a pointless combination of options. */+die("BUG: base=0 specified without NUM_BASE_SPECIFIER");+}elseif(base==16&&starts_with(number,"0x")){+/*+*Wewanttotreatthisaszeroterminatedby+*an'x',whereasstrtol()/strtoul()would+*silentlyeatthe"0x".Weaccomplishthis+*bytreatingitasabase10number:+*/+*flags=(*flags&~NUM_BASE_MASK)|10;+}+}+return0;+}++intparse_l(constchar*s,unsignedintflags,long*result,char**endptr)+{+longl;+constchar*end;+interr=0;++err=parse_precheck(s,&flags);+if(err)+returnerr;++/*+*Nowletstrtol()dotheheavylifting:+*/+errno=0;+l=strtol(s,(char**)&end,flags&NUM_BASE_MASK);+if(errno){+if(errno==ERANGE){+if(!(flags&NUM_SATURATE))+return-NUM_SATURATE;+}else{+return-NUM_OTHER_ERROR;+}+}+if(end==s)+return-NUM_NO_DIGITS;++if(*end&&!(flags&NUM_TRAILING))+return-NUM_TRAILING;++/* Everything was OK */+*result=l;+if(endptr)+*endptr=(char*)end;+return0;+}++intparse_ul(constchar*s,unsignedintflags,+unsignedlong*result,char**endptr)+{+unsignedlongul;+constchar*end;+interr=0;++err=parse_precheck(s,&flags);+if(err)+returnerr;++/*+*Nowletstrtoul()dotheheavylifting:+*/+errno=0;+ul=strtoul(s,(char**)&end,flags&NUM_BASE_MASK);+if(errno){+if(errno==ERANGE){+if(!(flags&NUM_SATURATE))+return-NUM_SATURATE;+}else{+return-NUM_OTHER_ERROR;+}+}+if(end==s)+return-NUM_NO_DIGITS;++/*+*strtoul(),perversely,acceptsnegativenumbers,converting+*themtothepositivenumberwiththesamebitpattern.We+*don'teverwantthat.+*/+if((flags&NUM_NEGATIVE)&&ul){+if(!(flags&NUM_SATURATE))+return-NUM_SATURATE;+ul=0;+}++if(*end&&!(flags&NUM_TRAILING))+return-NUM_TRAILING;++/* Everything was OK */+*result=ul;+if(endptr)+*endptr=(char*)end;+return0;+}++intparse_i(constchar*s,unsignedintflags,int*result,char**endptr)+{+longl;+interr;+char*end;++err=parse_l(s,flags,&l,&end);+if(err)+returnerr;++if((int)l==l)+*result=l;+elseif(!(flags&NUM_SATURATE))+return-NUM_SATURATE;+else+*result=(l<=0)?INT_MIN:INT_MAX;++if(endptr)+*endptr=end;++return0;+}++intparse_ui(constchar*s,unsignedintflags,unsignedint*result,char**endptr)+{+unsignedlongul;+interr;+char*end;++err=parse_ul(s,flags,&ul,&end);+if(err)+returnerr;++if((unsignedint)ul==ul)+*result=ul;+elseif(!(flags&NUM_SATURATE))+return-NUM_SATURATE;+else+*result=UINT_MAX;++if(endptr)+*endptr=end;++return0;+}
@@ -0,0 +1,207 @@+#ifndef NUMPARSE_H+#define NUMPARSE_H++/*+*Functionsforparsingintegralnumbers.+*+*strtol()andstrtoul()areveryflexible,infacttooflexiblefor+*manypurposes.Thesefunctionswrapthemtomakethemeasiertouse+*inastricterway.+*+*Therearetwoclassesoffunction,parse_*()andconvert_*().The+*formertrytoreadanumberfromthefrontofastringandreporta+*pointertothecharacterfollowingthenumber.Thelatterdon't+*reporttheendofthenumber,andaremeanttobeusedwhenthe+*inputstringshouldcontainonlyasinglenumber,withnotrailing+*characters.+*+*Eachclassoffunctionshasfourvariants:+*+*-parse_l(),convert_l()--parselongints+*-parse_ul(),convert_ul()--parseunsignedlongints+*-parse_i(),convert_i()--parseints+*-parse_ui(),convert_ui()--parseunsignedints+*+*Thestyleofparsingiscontrolledbyaflagsargumentwhich+*encodesboththebaseofthenumberandmanyotheroptions.The+*baseisencodedbyitsnumericalvalue(2<=base<=36),orzero+*ifitshouldbedeterminedautomaticallybasedonwhetherthe+*numberhasa"0x"or"0"prefix.+*+*Thefunctionsallreturnzeroonsuccess.Onerror,theyreturna+*negativeintegerindicatingthefirsterrorthatwasdetected.For+*example,ifnosigncharacterswereallowedbutthestring+*containeda'-',thefunctionwillreturn-NUM_MINUS.Ifthereis+*anykindoferror,*resultand*endptrareunchanged.+*+*Examples:+*+*-Converthexadecimalstringsintoanunsignedint.Dieifthere+*areanycharactersinsbesideshexadecimaldigits,orifthe+*resultexceedstherangeofanunsignedint:+*+*if(convert_ui(s,16,&result))+*die("...");+*+*-Readabase-tenlongnumberfromthefrontofastring,allowing+*signcharactersandsettingendptrtopointatanytrailing+*characters:+*+*if(parse_l(s,10|NUM_SIGN|NUM_TRAILING,&result,&endptr))+*die("...");+*+*-Convertdecimalstringsintoasignedint,butnotallowingthe+*stringtocontaina'+'or'-'prefix(andtherebyindirectly+*ensuringthattheresultwillbenon-negative):+*+*if(convert_i(s,10,&result))+*die("...");+*+*-Convertsintoasignedint,interpretingprefix"0x"tomean+*hexadecimaland"0"tomeanoctal.Ifthevaluedoesn'tfitinan+*unsignedint,setresulttoINT_MINorINT_MAX.+*+*if(convert_i(s,NUM_SLOPPY,&result))+*die("...");+*/+++/*+*Constantsforparsingnumbers.+*+*Thesecanbepassedinflagstoallowthespecifiedfeatures.Also,+*ifthereisanerrorparsinganumber,theparsingfunctionsreturn+*thenegatedvalueofoneoftheseconstants(orNUM_NO_DIGITSor+*NUM_OTHER_ERROR)toindicatethefirsterrordetected.+*/++/*+*Thelowest6bitsofflagsholdthenumericalbasethatshouldbe+*usedtoparsethenumber,2<=base<=36.Ifbaseissetto0,+*thenNUM_BASE_SPECIFIERmustbesettoo;inthiscase,thebaseis+*detectedautomaticallyfromthestring'sprefix.+*/+#define NUM_BASE_MASK 0x3f++/* Skip any whitespace before the number. */+#define NUM_LEADING_WHITESPACE (1 << 8)++/* Allow a leading '+'. */+#define NUM_PLUS (1 << 9)++/* Allow a leading '-'. */+#define NUM_MINUS (1 << 10)++/*+*Allowaleadingbasespecifier:+*-Ifbaseis0:aleading"0x"indicatesbase16;aleading"0"+*indicatesbase8;otherwise,assumebase10.+*-Ifbaseis16:aleading"0x"isallowedandskippedover.+*/+#define NUM_BASE_SPECIFIER (1 << 11)++/*+*Ifthenumberisnotintheallowedrange,returnthesmallestor+*largestrepresentablevalueinstead.+*/+#define NUM_SATURATE (1 << 12)++/*+*Justparseuntiltheendofthenumber,ignoringanysubsequent+*characters.Ifthisoptionisnotspecified,thenitisanerrorif+*thewholestringcannotbeparsed.+*/+#define NUM_TRAILING (1 << 13)+++/* Additional errors that can come from parsing numbers: */++/* There were no valid digits */+#define NUM_NO_DIGITS (1 << 14)+/* There was some other error reported by strtol()/strtoul(): */+#define NUM_OTHER_ERROR (1 << 15)++/*+*PleasenotethatthereisalsoaNUM_NEGATIVE,whichisused+*internally.+*/++/*+*Nowdefinesomeusefulcombinationsofparsingoptions:+*/++/* A bunch of digits with an optional sign. */+#define NUM_SIGN (NUM_PLUS | NUM_MINUS)++/*+*Beasliberalaspossiblewiththeformofthenumberitself+*(thoughifyoualsowanttoallowleadingwhitespaceand/or+*trailingcharacters,youshouldcombinethiswith+*NUM_LEADING_WHITESPACEand/orNUM_TRAILING).+*/+#define NUM_SLOPPY (NUM_SIGN | NUM_SATURATE | NUM_BASE_SPECIFIER)+++/*+*Numberparsingfunctions:+*+*Thefollowingfunctionsparseanumber(long,unsignedlong,int,+*orunsignedintrespectively)fromthefrontofs,storingthe+*valueto*resultandstoringapointertothefirstcharacterafter+*thenumberto*endptr.flagsspecifieshowthenumbershouldbe+*parsed,includingwhichbaseshouldbeused.flagsisacombination+*ofthenumericalbase(2-36)andtheNUM_*constantsabove(see).+*Return0onsuccessoranegativevalueiftherewasanerror.On+*failure,*resultand*entptrareleftunchanged.+*+*PleasenotethatifNUM_TRAILINGisnotset,thenitis+*neverthelessanerrorifthereareanycharactersbetweentheend+*ofthenumberandtheendofthestring.+*/++intparse_l(constchar*s,unsignedintflags,+long*result,char**endptr);++intparse_ul(constchar*s,unsignedintflags,+unsignedlong*result,char**endptr);++intparse_i(constchar*s,unsignedintflags,+int*result,char**endptr);++intparse_ui(constchar*s,unsignedintflags,+unsignedint*result,char**endptr);+++/*+*Numberconversionfunctions:+*+*Thefollowingfunctionsparseastringintoanumber.Theyare+*identicaltotheparse_*()functionsabove,exceptthattheendptr+*isnotreturned.Thesearemostusefulwhenparsingawholestring+*intoanumber;i.e.,when(flags&NUM_TRAILING)isunset.+*/+staticinlineintconvert_l(constchar*s,unsignedintflags,+long*result)+{+returnparse_l(s,flags,result,NULL);+}++staticinlineintconvert_ul(constchar*s,unsignedintflags,+unsignedlong*result)+{+returnparse_ul(s,flags,result,NULL);+}++staticinlineintconvert_i(constchar*s,unsignedintflags,+int*result)+{+returnparse_i(s,flags,result,NULL);+}++staticinlineintconvert_ui(constchar*s,unsignedintflags,+unsignedint*result)+{+returnparse_ui(s,flags,result,NULL);+}++#endif /* NUMPARSE_H */
From: Michael Haggerty <hidden> Date: 2016-06-15 23:04:11
Their callers have been changed to use the numparse module.
Signed-off-by: Michael Haggerty <redacted>
---
git-compat-util.h | 26 --------------------------
1 file changed, 26 deletions(-)
From: Michael Haggerty <hidden> Date: 2016-06-15 23:04:11
Use convert_ui() instead of strtoul_ui() when parsing tree entries'
modes. This tightens up the parsing a bit:
* Leading whitespace is no longer allowed
* '+' and '-' are no longer allowed
Signed-off-by: Michael Haggerty <redacted>
---
contrib/convert-objects/convert-objects.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
From: Michael Haggerty <hidden> Date: 2016-06-15 23:04:11
This adds error checking, where previously there was none. It also
disallows '+' and '-', leading whitespace, and trailing junk.
Signed-off-by: Michael Haggerty <redacted>
---
revision.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
From: Michael Haggerty <hidden> Date: 2016-06-15 23:04:11
Increment "arg" when a character is consumed, not just before consuming
the next character.
Signed-off-by: Michael Haggerty <redacted>
---
diff.c | 8 +++-----
1 file changed, 3 insertions(+), 5 deletions(-)
@@ -3358,14 +3358,13 @@ static int opt_arg(const char *arg, int arg_short, const char *arg_long, int *vacharc,*eq;intlen;-if(*arg!='-')+if(*arg++!='-')return0;-c=*++arg;+c=*arg++;if(!c)return0;if(c==arg_short){-c=*++arg;-if(!c)+if(!*arg)return1;/* optional argument was missing */if(convert_i(arg,10,val))die("The value for -%c must be a non-negative integer",arg_short);
@@ -3373,7 +3372,6 @@ static int opt_arg(const char *arg, int arg_short, const char *arg_long, int *va}if(c!='-')return0;-arg++;eq=strchrnul(arg,'=');len=eq-arg;if(!len||strncmp(arg,arg_long,len))
From: Michael Haggerty <hidden> Date: 2016-06-15 23:04:11
die() with an error message if the argument is not a non-negative
integer. This change tightens up parsing: '+' and '-', leading
whitespace, and trailing junk are all disallowed now.
Signed-off-by: Michael Haggerty <redacted>
---
diff.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
From: Michael Haggerty <hidden> Date: 2016-06-15 23:04:11
The previous code used strtoul() without any checks that it succeeded.
Instead use convert_l(), in strict mode, and die() if there is an
error. This tightens up the parsing:
* Leading whitespace is no longer allowed
* '+' and '-' are no longer allowed
* Trailing junk is not allowed
Signed-off-by: Michael Haggerty <redacted>
---
diff.c | 13 +++++++------
1 file changed, 7 insertions(+), 6 deletions(-)
@@ -2393,12 +2394,12 @@ static void builtin_diff(const char *name_a,xecfg.flags|=XDL_EMIT_FUNCCONTEXT;if(pe)xdiff_set_find_func(&xecfg,pe->pattern,pe->cflags);-if(!diffopts)-;-elseif(skip_prefix(diffopts,"--unified=",&v))-xecfg.ctxlen=strtoul(v,NULL,10);-elseif(skip_prefix(diffopts,"-u",&v))-xecfg.ctxlen=strtoul(v,NULL,10);+if(diffopts+&&(skip_prefix(diffopts,"--unified=",&v)||+skip_prefix(diffopts,"-u",&v))){+if(convert_l(v,10,&xecfg.ctxlen))+die("--unified argument must be a non-negative integer");+}if(o->word_diff)init_diff_words_data(&ecbdata,o,one,two);xdi_diff_outf(&mf1,&mf2,fn_out_consume,&ecbdata,
From: Michael Haggerty <hidden> Date: 2016-06-15 23:04:11
die() with an error message if the argument is not a non-negative
integer. This change tightens up parsing: '+' and '-', leading
whitespace, and trailing junk are all disallowed now.
Signed-off-by: Michael Haggerty <redacted>
---
diff.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
From: Michael Haggerty <hidden> Date: 2016-06-15 23:04:11
This shortens the code and avoids the old code's careless truncation
from unsigned long to int.
Signed-off-by: Michael Haggerty <redacted>
---
diff.c | 28 ++++++++--------------------
1 file changed, 8 insertions(+), 20 deletions(-)
@@ -3366,16 +3366,10 @@ static int opt_arg(const char *arg, int arg_short, const char *arg_long, int *vaif(c==arg_short){c=*++arg;if(!c)-return1;-if(isdigit(c)){-char*end;-intn=strtoul(arg,&end,10);-if(*end)-return0;-*val=n;-return1;-}-return0;+return1;/* optional argument was missing */+if(convert_i(arg,10,val))+return0;+return1;}if(c!='-')return0;
@@ -3384,16 +3378,10 @@ static int opt_arg(const char *arg, int arg_short, const char *arg_long, int *valen=eq-arg;if(!len||strncmp(arg,arg_long,len))return0;-if(*eq){-intn;-char*end;-if(!isdigit(*++eq))-return0;-n=strtoul(eq,&end,10);-if(*end)-return0;-*val=n;-}+if(!*eq)+return1;/* '=' and optional argument were missing */+if(convert_i(eq+1,10,val))+return0;return1;}
From: Michael Haggerty <hidden> Date: 2016-06-15 23:04:11
If an argument is there, but it can't be parsed as a non-positive
number, then die() rather than returning 0.
Signed-off-by: Michael Haggerty <redacted>
---
diff.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
@@ -3368,7 +3368,7 @@ static int opt_arg(const char *arg, int arg_short, const char *arg_long, int *vaif(!c)return1;/* optional argument was missing */if(convert_i(arg,10,val))-return0;+die("The value for -%c must be a non-negative integer",arg_short);return1;}if(c!='-')
@@ -3381,7 +3381,7 @@ static int opt_arg(const char *arg, int arg_short, const char *arg_long, int *vaif(!*eq)return1;/* '=' and optional argument were missing */if(convert_i(eq+1,10,val))-return0;+die("The value for --%s must be a non-negative integer",arg_long);return1;}
From: Michael Haggerty <hidden> Date: 2016-06-15 23:04:11
opt_arg() is never called with val set to NULL, so remove the code for
handling that eventuality (which anyway wasn't handled consistently in
the function).
Signed-off-by: Michael Haggerty <redacted>
---
diff.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
On Tue, Mar 17, 2015 at 11:00 PM, Michael Haggerty [off-list ref] wrote:
Michael Haggerty (14):
numparse: new module for parsing integral numbers
cacheinfo_callback(): use convert_ui() when handling "--cacheinfo"
write_subdirectory(): use convert_ui() for parsing mode
handle_revision_opt(): use skip_prefix() in many places
handle_revision_opt(): use convert_i() when handling "-<digit>"
strtoul_ui(), strtol_i(): remove functions
handle_revision_opt(): use convert_ui() when handling "--abbrev="
builtin_diff(): detect errors when parsing --unified argument
opt_arg(): val is always non-NULL
opt_arg(): use convert_i() in implementation
opt_arg(): report errors parsing option values
opt_arg(): simplify pointer handling
diff_opt_parse(): use convert_i() when handling "-l<num>"
diff_opt_parse(): use convert_i() when handling --abbrev=<num>
Thank you for doing it. I was about to write another number parser and
you did it :D Maybe you can add another patch to convert the only
strtol in upload-pack.c to parse_ui. This place should accept positive
number in base 10, plus sign is not accepted.
--
Duy
From: Michael Haggerty <hidden> Date: 2016-06-15 23:04:11
On 03/18/2015 12:05 AM, Duy Nguyen wrote:
On Tue, Mar 17, 2015 at 11:00 PM, Michael Haggerty [off-list ref] wrote:
quoted
Michael Haggerty (14):
numparse: new module for parsing integral numbers
cacheinfo_callback(): use convert_ui() when handling "--cacheinfo"
write_subdirectory(): use convert_ui() for parsing mode
handle_revision_opt(): use skip_prefix() in many places
handle_revision_opt(): use convert_i() when handling "-<digit>"
strtoul_ui(), strtol_i(): remove functions
handle_revision_opt(): use convert_ui() when handling "--abbrev="
builtin_diff(): detect errors when parsing --unified argument
opt_arg(): val is always non-NULL
opt_arg(): use convert_i() in implementation
opt_arg(): report errors parsing option values
opt_arg(): simplify pointer handling
diff_opt_parse(): use convert_i() when handling "-l<num>"
diff_opt_parse(): use convert_i() when handling --abbrev=<num>
Thank you for doing it. I was about to write another number parser and
you did it :D Maybe you can add another patch to convert the only
strtol in upload-pack.c to parse_ui. This place should accept positive
number in base 10, plus sign is not accepted.
If the general direction of this patch series is accepted, I'll
gradually try to go through the codebase, replacing *all* integer
parsing with these functions. So there's no need to request particular
callers of strtol()/strtoul() to be converted; I'll get to them all
sooner or later (I hope).
But in case you have some reason that you want upload-pack.c to be
converted right away, I just pushed that change (plus some related
cleanups) to my GitHub repo [1]. The branch depends only on the first
patch of the "numparse" patch series.
By the way, some other packet line parsing code in that file doesn't
verify that there are no trailing characters on the lines that they
process. That might be another thing that should be tightened up.
Michael
[1] https://github.com/mhagger/git.git, branch "upload-pack-numparse"
--
Michael Haggerty
mhagger@alum.mit.edu
On Wed, Mar 18, 2015 at 4:47 PM, Michael Haggerty [off-list ref] wrote:
quoted
Thank you for doing it. I was about to write another number parser and
you did it :D Maybe you can add another patch to convert the only
strtol in upload-pack.c to parse_ui. This place should accept positive
number in base 10, plus sign is not accepted.
If the general direction of this patch series is accepted, I'll
gradually try to go through the codebase, replacing *all* integer
parsing with these functions. So there's no need to request particular
callers of strtol()/strtoul() to be converted; I'll get to them all
sooner or later (I hope).
Good to know. No there's no hurry in converting this particular place.
It's just tightening things up, not bug fixing or anything.
--
Duy
From: Jeff King <hidden> Date: 2016-06-15 23:04:11
On Wed, Mar 18, 2015 at 10:47:40AM +0100, Michael Haggerty wrote:
But in case you have some reason that you want upload-pack.c to be
converted right away, I just pushed that change (plus some related
cleanups) to my GitHub repo [1]. The branch depends only on the first
patch of the "numparse" patch series.
By the way, some other packet line parsing code in that file doesn't
verify that there are no trailing characters on the lines that they
process. That might be another thing that should be tightened up.
Do you mean that upload-pack gets a pkt-line of length N that contains a
line of length M, and then doesn't check that M==N? We use the space
between M and N for passing capabilities and other metadata around.
Or do you mean that we see lines like:
want [0-9a-f]{40} ...\n
and do not bother looking at the "..." that comes after the data we
expect? That I can believe, and I don't think it would hurt to tighten
up (we shouldn't need it for extensibility, as anybody trying to stick
extra data there should do so only after using a capability flag earlier
in the protocol).
-Peff
From: Michael Haggerty <hidden> Date: 2016-06-15 23:04:11
On 03/18/2015 11:03 AM, Jeff King wrote:
On Wed, Mar 18, 2015 at 10:47:40AM +0100, Michael Haggerty wrote:
quoted
But in case you have some reason that you want upload-pack.c to be
converted right away, I just pushed that change (plus some related
cleanups) to my GitHub repo [1]. The branch depends only on the first
patch of the "numparse" patch series.
By the way, some other packet line parsing code in that file doesn't
verify that there are no trailing characters on the lines that they
process. That might be another thing that should be tightened up.
Do you mean that upload-pack gets a pkt-line of length N that contains a
line of length M, and then doesn't check that M==N? We use the space
between M and N for passing capabilities and other metadata around.
Or do you mean that we see lines like:
want [0-9a-f]{40} ...\n
and do not bother looking at the "..." that comes after the data we
expect? That I can believe, and I don't think it would hurt to tighten
up (we shouldn't need it for extensibility, as anybody trying to stick
extra data there should do so only after using a capability flag earlier
in the protocol).
Perhaps use a /* one-line style comment */ to reduce vertical space
consumption a bit, thus make it (very slightly) easier to run the eye
over the code.
Would it reduce cognitive load slightly (and reduce vertical space
consumption) to rephrase the conditionals as:
if (errno == ERANGE && !(flags & NUM_SATURATE))
return -NUM_SATURATE;
if (errno && errno != ERANGE)
return -NUM_OTHER_ERROR;
or something similar?
More below.
quoted hunk
+ if (end == s)
+ return -NUM_NO_DIGITS;
+
+ if (*end && !(flags & NUM_TRAILING))
+ return -NUM_TRAILING;
+
+ /* Everything was OK */
+ *result = l;
+ if (endptr)
+ *endptr = (char *)end;
+ return 0;
+}
+ *
+ * if (convert_i(s, NUM_SLOPPY, &result))
+ * die("...");
+ */
+
+/*
+ * The lowest 6 bits of flags hold the numerical base that should be
+ * used to parse the number, 2 <= base <= 36. If base is set to 0,
+ * then NUM_BASE_SPECIFIER must be set too; in this case, the base is
+ * detected automatically from the string's prefix.
Does this restriction go against the goal of making these functions
convenient, even while remaining strict? Is there a strong reason for
not merely inferring NUM_BASE_SPECIFIER when base is 0? Doing so would
make it consistent with strto*l() without (I think) introducing any
ambiguity.
+ */
+/*
+ * Number parsing functions:
+ *
+ * The following functions parse a number (long, unsigned long, int,
+ * or unsigned int respectively) from the front of s, storing the
+ * value to *result and storing a pointer to the first character after
+ * the number to *endptr. flags specifies how the number should be
+ * parsed, including which base should be used. flags is a combination
+ * of the numerical base (2-36) and the NUM_* constants above (see).
"(see)" what?
+ * Return 0 on success or a negative value if there was an error. On
+ * failure, *result and *entptr are left unchanged.
+ *
+ * Please note that if NUM_TRAILING is not set, then it is
+ * nevertheless an error if there are any characters between the end
+ * of the number and the end of the string.
Again, on the subject of convenience, why this restriction? The stated
purpose of the parse_*() functions is to parse the number from the
front of the string and return a pointer to the first non-numeric
character following. As a reader of this API, I interpret that as
meaning that NUM_TRAILING is implied. Is there a strong reason for not
inferring NUM_TRAILING for the parse_*() functions at the API level?
(I realize that the convert_*() functions are built atop parse_*(),
but that's an implementation detail.)
+ */
+
+int parse_l(const char *s, unsigned int flags,
+ long *result, char **endptr);
Do we want to perpetuate the ugly (char **) convention for 'endptr'
from strto*l()? Considering that the incoming string is const, it
seems undesirable to return a non-const pointer to some place inside
that string.
+/*
+ * Number conversion functions:
+ *
+ * The following functions parse a string into a number. They are
+ * identical to the parse_*() functions above, except that the endptr
+ * is not returned. These are most useful when parsing a whole string
+ * into a number; i.e., when (flags & NUM_TRAILING) is unset.
I can formulate arguments for allowing or disallowing NUM_TRAILING
with convert_*(), however, given their purpose of parsing the entire
string into a number, as a reader of the API, I would kind of expect
the convert_*() functions to ensure that NUM_TRAILING is not set
(either by forcibly clearing it or erroring out as an inconsistent
request if it is set).
+ */
+static inline int convert_l(const char *s, unsigned int flags,
+ long *result)
+{
+ return parse_l(s, flags, result, NULL);
+}
Perhaps use a /* one-line style comment */ to reduce vertical space
consumption a bit, thus make it (very slightly) easier to run the eye
over the code.
Would it reduce cognitive load slightly (and reduce vertical space
consumption) to rephrase the conditionals as:
if (errno == ERANGE && !(flags & NUM_SATURATE))
return -NUM_SATURATE;
if (errno && errno != ERANGE)
return -NUM_OTHER_ERROR;
or something similar?
The most common case by far should be that errno is zero. The code as
written only needs one test for that case, whereas your code needs two
tests. I think it is worth compromising on code clarity a tiny bit to
avoid the extra test.
More below.
quoted
+ if (end == s)
+ return -NUM_NO_DIGITS;
+
+ if (*end && !(flags & NUM_TRAILING))
+ return -NUM_TRAILING;
+
+ /* Everything was OK */
+ *result = l;
+ if (endptr)
+ *endptr = (char *)end;
+ return 0;
+}
+ *
+ * if (convert_i(s, NUM_SLOPPY, &result))
+ * die("...");
+ */
+
+/*
+ * The lowest 6 bits of flags hold the numerical base that should be
+ * used to parse the number, 2 <= base <= 36. If base is set to 0,
+ * then NUM_BASE_SPECIFIER must be set too; in this case, the base is
+ * detected automatically from the string's prefix.
Does this restriction go against the goal of making these functions
convenient, even while remaining strict? Is there a strong reason for
not merely inferring NUM_BASE_SPECIFIER when base is 0? Doing so would
make it consistent with strto*l() without (I think) introducing any
ambiguity.
I thought about doing this. If it were possible to eliminate
NUM_BASE_SPECIFIER altogether, then there is no doubt that it would be a
good change. But NUM_BASE_SPECIFIER also has an effect when base==16;
namely, that an "0x" prefix, if present, is consumed. So
parse_i("0xff", 16 | NUM_BASE_SPECIFIER, &result, &endptr)
gives result==255 and endptr==s+4, whereas
parse_i("0xff", 16, &result, &endptr)
gives result==0 and entptr==s+1 (it treats the "x" as the end of the
string).
We could forgo that feature, effectively allowing a base specifier if
and only if base==0. But I didn't want to rule out allowing an optional
base specifier for base==16, in which case NUM_BASE_SPECIFIER can't be
dispensed with entirely.
If you agree with that, then the remaining question is: which policy is
less error-prone? My thinking was that forcing the caller to specify
NUM_BASE_SPECIFIER explicitly when they select base==0 will serve as a
reminder that the two features are intertwined. Because another
imaginable policy (arguably more consistent with the policy for base!=0)
would be that
convert_i(s, 0, &result)
, because it *doesn't* specify NUM_BASE_SPECIFIER, doesn't allow a base
prefix, and therefore indirectly only allows base-10 numbers.
But I don't feel strongly about this.
quoted
+ */
+/*
+ * Number parsing functions:
+ *
+ * The following functions parse a number (long, unsigned long, int,
+ * or unsigned int respectively) from the front of s, storing the
+ * value to *result and storing a pointer to the first character after
+ * the number to *endptr. flags specifies how the number should be
+ * parsed, including which base should be used. flags is a combination
+ * of the numerical base (2-36) and the NUM_* constants above (see).
"(see)" what?
That was meant to be a pointer to the documentation for the NUM_*
constants. But obviously it was too oblique, so I will make it more
explicit, maybe
... and the NUM_* constants documented above.
quoted
+ * Return 0 on success or a negative value if there was an error. On
+ * failure, *result and *entptr are left unchanged.
+ *
+ * Please note that if NUM_TRAILING is not set, then it is
+ * nevertheless an error if there are any characters between the end
+ * of the number and the end of the string.
Again, on the subject of convenience, why this restriction? The stated
purpose of the parse_*() functions is to parse the number from the
front of the string and return a pointer to the first non-numeric
character following. As a reader of this API, I interpret that as
meaning that NUM_TRAILING is implied. Is there a strong reason for not
inferring NUM_TRAILING for the parse_*() functions at the API level?
(I realize that the convert_*() functions are built atop parse_*(),
but that's an implementation detail.)
Yes, I'd also thought about that change:
* Make NUM_TRAILING private.
* Make the current parse_*() functions private.
* Add new parse_*(s, flags, result, endptr) functions that imply
NUM_TRAILING (and maybe they should *require* a non-null endptr argument).
* Change the convert_*() to not allow the NUM_TRAILING flag.
This would add a little bit of code, so I didn't do it originally. But
since you seem to like the idea too, I guess I will make the change.
quoted
+ */
+
+int parse_l(const char *s, unsigned int flags,
+ long *result, char **endptr);
Do we want to perpetuate the ugly (char **) convention for 'endptr'
from strto*l()? Considering that the incoming string is const, it
seems undesirable to return a non-const pointer to some place inside
that string.
Yes, I guess we could do some internal casting and expose endptr as
(const char **). It would make it a bit harder if the user actually
wants to pass a non-const string to the function and then modify the
string via the returned endptr, but I haven't run into that pattern yet
so let's make the change you suggested.
quoted
+/*
+ * Number conversion functions:
+ *
+ * The following functions parse a string into a number. They are
+ * identical to the parse_*() functions above, except that the endptr
+ * is not returned. These are most useful when parsing a whole string
+ * into a number; i.e., when (flags & NUM_TRAILING) is unset.
I can formulate arguments for allowing or disallowing NUM_TRAILING
with convert_*(), however, given their purpose of parsing the entire
string into a number, as a reader of the API, I would kind of expect
the convert_*() functions to ensure that NUM_TRAILING is not set
(either by forcibly clearing it or erroring out as an inconsistent
request if it is set).
Yes, I think I will change this, as discussed above.
quoted
+ */
+static inline int convert_l(const char *s, unsigned int flags,
+ long *result)
+{
+ return parse_l(s, flags, result, NULL);
+}
Thanks for your careful and thoughtful comments!
Michael
--
Michael Haggerty
mhagger@alum.mit.edu
From: Jeff King <hidden> Date: 2016-06-15 23:04:12
On Tue, Mar 17, 2015 at 05:00:02PM +0100, Michael Haggerty wrote:
My main questions:
* Do people like the API? My main goal was to make these functions as
painless as possible to use correctly, because there are so many
call sites.
* Is it too gimmicky to encode the base together with other options in
`flags`? (I think it is worth it to avoid the need for another
parameter, which callers could easily put in the wrong order.)
I definitely like the overall direction of this. My first thought was
that it does seem like there are a lot of possible options to the
functions (and OR-ing the flags with the base does seem weird, though I
can't think of a plausible case where it would actually cause errors).
Many of those options don't seem used in the example conversions (I'm
not clear who would want NUM_SATURATE, for example).
I wondered if we could do away with the radix entirely. Wouldn't we be
asking for base 10 most of the time? Of course, your first few patches
show octal parsing, but I wonder if we should actually have a separate
parse_mode() for that, since that seems to be the general reason for
doing octal parsing. 100000644 does not overflow an int, but it is
hardly a reasonable mode.
I also wondered if we could get rid of NUM_SIGN in favor of just having
the type imply it (e.g., convert_l would always allow negative numbers,
whereas convert_ul would not). But I suppose there are times when we end
up using an "int" to store an unsigned value for a good reason (e.g.,
"-1" is a sentinel value, but we expect only positive values from the
user). So that might be a bad idea.
I notice that you go up to "unsigned long" here for sizes. If we want to
use this everywhere, we'll need something larger for parsing off_t
values on 32-bit machines (though I would not at all be surprised with
the current code if 32-bit machines have a hard time configuring a
pack.packSizeLimit above 4G).
I wonder how much of the boilerplate in the parse_* functions could be
factored out to use a uintmax_t, with the caller just providing the
range. That would make it easier to add new types like off_t, and
possibly even constrained types (e.g., an integer from 0 to 100). On the
other hand, you mentioned to me elsewhere that there may be some bugs in
the range-checks of config.c's integer parsing. I suspect they are
related to exactly this kind of refactoring, so perhaps writing things
out is best.
* Am I making callers too strict? In many cases where a positive
integer is expected (e.g., "--abbrev=<num>"), I have been replacing
code like
[...]
IMHO most of the tightening happening here is a good thing, and means we
are more likely to notice mistakes rather than silently doing something
stupid.
For sites that currently allow it, I could imagine people using hex
notation for some values, though, depending on the context. It looks
there aren't many of them ((it is just when the radix is "0", right?).
Some of them look to be accidental (does anybody really ask for
--threads=0x10?), but others might not be (the pack index-version
contains an offset field that might be quite big).
Feel free to ignore any or all of that. It is not so much criticism as
a dump of thoughts I had while reading the patches. Perhaps you can pick
something useful out of that, and perhaps not. :)
-Peff
From: Eric Sunshine <hidden> Date: 2016-06-15 23:04:13
On Wed, Mar 18, 2015 at 6:47 PM, Michael Haggerty [off-list ref] wrote:
On 03/18/2015 07:27 PM, Eric Sunshine wrote:
quoted
On Tuesday, March 17, 2015, Michael Haggerty [off-list ref] wrote:
quoted
Implement wrappers for strtol() and strtoul() that are safer and more
convenient to use.
+ * The lowest 6 bits of flags hold the numerical base that should be
+ * used to parse the number, 2 <= base <= 36. If base is set to 0,
+ * then NUM_BASE_SPECIFIER must be set too; in this case, the base is
+ * detected automatically from the string's prefix.
Does this restriction go against the goal of making these functions
convenient, even while remaining strict? Is there a strong reason for
not merely inferring NUM_BASE_SPECIFIER when base is 0? Doing so would
make it consistent with strto*l() without (I think) introducing any
ambiguity.
I thought about doing this. If it were possible to eliminate
NUM_BASE_SPECIFIER altogether, then there is no doubt that it would be a
good change. But NUM_BASE_SPECIFIER also has an effect when base==16;
namely, that an "0x" prefix, if present, is consumed. So
parse_i("0xff", 16 | NUM_BASE_SPECIFIER, &result, &endptr)
gives result==255 and endptr==s+4, whereas
parse_i("0xff", 16, &result, &endptr)
gives result==0 and entptr==s+1 (it treats the "x" as the end of the
string).
We could forgo that feature, effectively allowing a base specifier if
and only if base==0. But I didn't want to rule out allowing an optional
base specifier for base==16, in which case NUM_BASE_SPECIFIER can't be
dispensed with entirely.
Making base==0 a special case doesn't have to mean ruling out or
eliminating NUM_BASE_SPECIFIER for the base==16 case. However, a
special case base==0 would make the API a bit non-orthogonal and
require extra documentation. But for those familiar with how strto*l()
treats base==0 specially, the rule of least surprise may apply.
If you agree with that, then the remaining question is: which policy is
less error-prone? My thinking was that forcing the caller to specify
NUM_BASE_SPECIFIER explicitly when they select base==0 will serve as a
reminder that the two features are intertwined.
Since base==0 would unambiguously imply NUM_BASE_SPECIFIER, being able
to tersely say convert_i(s, 0, &result) would be a win from a
convenience perspective. However...
Because another
imaginable policy (arguably more consistent with the policy for base!=0)
would be that
convert_i(s, 0, &result)
, because it *doesn't* specify NUM_BASE_SPECIFIER, doesn't allow a base
prefix, and therefore indirectly only allows base-10 numbers.
But I don't feel strongly about this.
I don't feel strongly about it either, and can formulate arguments
either way. Assuming you stick with your current design, if the
strictness of having to specify NUM_BASE_SPECIFIER for base==0 proves
too burdensome, it can be loosened later by making base==0 a special
case.
On the other hand, if you go with Junio's suggestion of choosing names
for these functions which more closely mirror strto*l() names, then
the rule of least surprise might suggest that a special case for
base==0 has merit.
quoted
quoted
+ * Return 0 on success or a negative value if there was an error. On
+ * failure, *result and *entptr are left unchanged.
+ *
+ * Please note that if NUM_TRAILING is not set, then it is
+ * nevertheless an error if there are any characters between the end
+ * of the number and the end of the string.
Again, on the subject of convenience, why this restriction? The stated
purpose of the parse_*() functions is to parse the number from the
front of the string and return a pointer to the first non-numeric
character following. As a reader of this API, I interpret that as
meaning that NUM_TRAILING is implied. Is there a strong reason for not
inferring NUM_TRAILING for the parse_*() functions at the API level?
(I realize that the convert_*() functions are built atop parse_*(),
but that's an implementation detail.)
Yes, I'd also thought about that change:
* Make NUM_TRAILING private.
* Make the current parse_*() functions private.
* Add new parse_*(s, flags, result, endptr) functions that imply
NUM_TRAILING (and maybe they should *require* a non-null endptr argument).
* Change the convert_*() to not allow the NUM_TRAILING flag.
This would add a little bit of code, so I didn't do it originally. But
since you seem to like the idea too, I guess I will make the change.
The other option (as Junio also suggested) is to collapse this API
into just the four parse_*() functions. All of the call-sites you
converted in this series invoked convert_*(), but it's not clear that
having two sets of functions which do almost the same thing is much of
a win. If the default is for NUM_TRAILING to be off, and if NULL
'endptr' is allowed, then:
parse_ul(s, 10, &result, NULL);
doesn't seem particularly burdensome compared with:
convert_ul(s, 10, &result);
A more compact API, with only the parse_*() functions, means less to
remember and less to document, and often is more orthogonal.
The argument for dropping the convert_*() functions is strengthened if
you follow Junio's suggestion to name these functions after the
strto*l() variations.
quoted
quoted
+int parse_l(const char *s, unsigned int flags,
+ long *result, char **endptr);
Do we want to perpetuate the ugly (char **) convention for 'endptr'
from strto*l()? Considering that the incoming string is const, it
seems undesirable to return a non-const pointer to some place inside
that string.
Yes, I guess we could do some internal casting and expose endptr as
(const char **). It would make it a bit harder if the user actually
wants to pass a non-const string to the function and then modify the
string via the returned endptr, but I haven't run into that pattern yet
so let's make the change you suggested.
I don't feel strongly about this either. It always struck me as a bit
of an API wart, but I can see the convenience value of non-const
'endptr' if the caller intends to modify the string. If you rename
these functions to mirror strto*l(), then the rule of least surprise
would suggest that 'endptr' should remain non-const.
From: Junio C Hamano <hidden> Date: 2016-06-15 23:04:13
Michael Haggerty [off-list ref] writes:
+static int parse_precheck(const char *s, unsigned int *flags)
+{
+ const char *number;
+
+ if (isspace(*s)) {
+ if (!(*flags & NUM_LEADING_WHITESPACE))
+ return -NUM_LEADING_WHITESPACE;
+ do {
+ s++;
+ } while (isspace(*s));
+ }
+
+ if (*s == '+') {
+ if (!(*flags & NUM_PLUS))
+ return -NUM_PLUS;
+ number = s + 1;
+ *flags &= ~NUM_NEGATIVE;
+ } else if (*s == '-') {
+ if (!(*flags & NUM_MINUS))
+ return -NUM_MINUS;
+ number = s + 1;
+ *flags |= NUM_NEGATIVE;
+ } else {
+ number = s;
+ *flags &= ~NUM_NEGATIVE;
+ }
+
+ if (!(*flags & NUM_BASE_SPECIFIER)) {
+ int base = *flags & NUM_BASE_MASK;
+ if (base == 0) {
+ /* This is a pointless combination of options. */
+ die("BUG: base=0 specified without NUM_BASE_SPECIFIER");
+ } else if (base == 16 && starts_with(number, "0x")) {
+ /*
+ * We want to treat this as zero terminated by
+ * an 'x', whereas strtol()/strtoul() would
+ * silently eat the "0x". We accomplish this
+ * by treating it as a base 10 number:
+ */
+ *flags = (*flags & ~NUM_BASE_MASK) | 10;
+ }
+ }
+ return 0;
+}
I somehow feel that a pre-processing that only _inspects_ part of
the string, without munging that string (e.g. notice '-' but feed
that to underlying strtol(3)) somewhat a brittle approach. When I
read the above for the first time, I somehow expected that the code
would notice leading '-', strip that leading '-' and remember the
fact that it did so in the *flags, let the strtol(3) to parse the
remainder _and_ always make sure the returned result is not negative
(because that would imply that the original input had two leading
minuses and digits), and give the sign based on what this preprocess
found out in *flags, and then seeing that there is no sign of such
processing in the caller I scratched my head.
I still have not convinced myself that what I am seeing in the
base==16 part in the above is correct.
From: Michael Haggerty <hidden> Date: 2016-06-15 23:04:16
On 03/19/2015 06:26 AM, Jeff King wrote:
On Tue, Mar 17, 2015 at 05:00:02PM +0100, Michael Haggerty wrote:
quoted
My main questions:
* Do people like the API? My main goal was to make these functions as
painless as possible to use correctly, because there are so many
call sites.
* Is it too gimmicky to encode the base together with other options in
`flags`? (I think it is worth it to avoid the need for another
parameter, which callers could easily put in the wrong order.)
I definitely like the overall direction of this. My first thought was
that it does seem like there are a lot of possible options to the
functions (and OR-ing the flags with the base does seem weird, though I
can't think of a plausible case where it would actually cause errors).
Many of those options don't seem used in the example conversions (I'm
not clear who would want NUM_SATURATE, for example).
There are a lot of options, but so far only few of them have been used.
As the call sites are rewritten we will see which of these features are
useful, and which can be dispensed with. If groups of options tend to be
used together, we can define constants for them like I've done with
NUM_SLOPPY and NUM_SIGN.
Regarding NUM_SATURATE: I'm not sure who would want it either, but I
thought there might be places where the user wants to specify
(effectively) "infinity", and it might be convenient to let him specify
something easy to type like "--max=999999999999" rather than
"--max=2147483647".
I wondered if we could do away with the radix entirely. Wouldn't we be
asking for base 10 most of the time? Of course, your first few patches
show octal parsing, but I wonder if we should actually have a separate
parse_mode() for that, since that seems to be the general reason for
doing octal parsing. 100000644 does not overflow an int, but it is
hardly a reasonable mode.
Again, as a first pass I wanted to just have a really flexible API so
that call sites can be rewritten without a lot of extra thought. If
somebody wants to add a parse_mode() function, it will be easy to build
on top of convert_ui(). But that change can be done after this one.
I also wondered if we could get rid of NUM_SIGN in favor of just having
the type imply it (e.g., convert_l would always allow negative numbers,
whereas convert_ul would not). But I suppose there are times when we end
up using an "int" to store an unsigned value for a good reason (e.g.,
"-1" is a sentinel value, but we expect only positive values from the
user). So that might be a bad idea.
Yes, as I was rewriting call sites, I found many that used the unsigned
variants of the parsing functions but stored the result in an int.
Probably some of these use -1 to denote "unset"; it might be that there
are other cases where the variable could actually be declared to be
"unsigned int".
Prohibiting signs when parsing signed quantities isn't really elegant
from an API purity point of view, but it sure is handy!
I notice that you go up to "unsigned long" here for sizes. If we want to
use this everywhere, we'll need something larger for parsing off_t
values on 32-bit machines (though I would not at all be surprised with
the current code if 32-bit machines have a hard time configuring a
pack.packSizeLimit above 4G).
Yes, probably. I haven't run into such call sites yet.
I wonder how much of the boilerplate in the parse_* functions could be
factored out to use a uintmax_t, with the caller just providing the
range. That would make it easier to add new types like off_t, and
possibly even constrained types (e.g., an integer from 0 to 100). On the
other hand, you mentioned to me elsewhere that there may be some bugs in
the range-checks of config.c's integer parsing. I suspect they are
related to exactly this kind of refactoring, so perhaps writing things
out is best.
It's not a lot of code yet. If we find out we need variants for size_t
and off_t and uintmax_t and intmax_t then such a refactoring would
definitely be worth considering.
quoted
* Am I making callers too strict? In many cases where a positive
integer is expected (e.g., "--abbrev=<num>"), I have been replacing
code like
[...]
IMHO most of the tightening happening here is a good thing, and means we
are more likely to notice mistakes rather than silently doing something
stupid.
For sites that currently allow it, I could imagine people using hex
notation for some values, though, depending on the context. It looks
there aren't many of them ((it is just when the radix is "0", right?).
Some of them look to be accidental (does anybody really ask for
--threads=0x10?), but others might not be (the pack index-version
contains an offset field that might be quite big).
Yes, we can even debate whether we want to implement a general policy
that user-entered integers can be specified in any radix. Probably
nobody will ever specify "--threads=0x10", but is there harm in allowing it?
Against the gain in flexibility, I see the following potential
disadvantages:
* Added cognitive burden for rarely-used flexibility.
* Other Git clients might feel obliged to be just as flexible, causing
their implementers extra work.
* Users might see commands written in unfamiliar ways (e.g. in scripts
or stackoverflow) and get confused.
* Octal is ambiguous with decimal. What to make of "--count=010"? People
who expect octal numbers to be allowed will think it means 8, whereas
people who don't expect octal numbers to be allowed (or don't even know
what an octal number is!) will think that it means 10. Such uses might
even already exist and have their meaning changed if we start allowing
octal.
All in all, I thought it is less error-prone to default to allowing only
decimal, except in selected situations where hex and octal are
traditionally used (e.g., file modes, memory limits).
Most of the call sites so far have explicitly specified decimal parsing,
and I have left them unchanged.
Feel free to ignore any or all of that. It is not so much criticism as
a dump of thoughts I had while reading the patches. Perhaps you can pick
something useful out of that, and perhaps not. :)
Thanks very much for the feedback!
Michael
--
Michael Haggerty
mhagger@alum.mit.edu