Since everyone's getting in on the C99 fun.
Well, $subject and a bit more. This RFC series has bits and pieces
from thing I've submitted before. I'd proposed to make variadic macros
a hard dependency before in [1] because I wanted to get to the goal in
$subject, perhaps the whole thing will be more convincing.
This also includes the die_message() in a recent series of mine[2]
that I abandoned.
At the end of this series we expose a config variable to have
usage/die/warning emit line numbers. I.e. going from:
$ git -c core.usageAddSource=false -c core.x=y config --get --bool core.x
fatal: bad boolean config value 'y' for 'core.x'
To:
$ git -c core.usageAddSource=false -c core.x=y config --get --bool core.x
fatal: config.c:1241: bad boolean config value 'y' for 'core.x'
I find that to make tracing down errors in the test suite, and 21/21
has a GIT_TEST_* mode to turn it on there (which fails a lot now, but
I'm hoping I'll eventually get passing).
But most importantly we've now got meaningful file/line numbers in
trace2 error events. I.e. from all of them being some line in usage.c:
$ GIT_TRACE2_EVENT=/dev/stdout ~/g/git/git -c core.usageAddSource=false -c core.x=y config --get --bool core.x 2>&1 2>/dev/null|grep error | jq -r .
{
"event": "error",
"sid": "20211115T221343.534151Z-Hc2f5b994-P003f3980",
"thread": "main",
"time": "2021-11-15T22:13:43.537981Z",
"file": "usage.c",
"line": 65,
"msg": "bad boolean config value 'y' for 'core.x'",
"fmt": "bad boolean config value '%s' for '%s'"
}
To:
$ GIT_TRACE2_EVENT=/dev/stdout ~/g/git/git -c core.usageAddSource=false -c core.x=y config --get --bool core.x 2>&1 2>/dev/null|grep error | jq -r .
{
"event": "error",
"sid": "20211115T221357.083824Z-Hc2f5b994-P003f4a82",
"thread": "main",
"time": "2021-11-15T22:13:57.087596Z",
"file": "config.c",
"line": 1241,
"msg": "bad boolean config value 'y' for 'core.x'",
"fmt": "bad boolean config value '%s' for '%s'"
}
I've got some speculation in 19/21 that this may make the "fmt" part
redundant, i.e. did we only add that because we couldn't group these
by file/line, but as noted there there's still some use-cases for
"fmt" even with this series. In any case, this series doesn't touch
that "fmt" key at all.
This is "RFC" mainly because there's a CI failure in 0061.2 with this,
I still can't figure out what that's about (or if it's some fluke
unrelated to this topic), but that has to be investigated.
But I wanted to see if people found the general idea interesting
too. I picked the CC list mainly from paging through "--grep=trace2",
and people who'd modified the tricker bits of usage.c code being
modified here.
1. https://lore.kernel.org/git/cover-0.2-00000000000-20210412T105422Z-avarab@gmail.com/
2. https://lore.kernel.org/git/cover-v3-0.6-00000000000-20211022T175227Z-avarab@gmail.com/
3. https://github.com/avar/git/runs/4216916706?check_suite_focus=true
Ævar Arnfjörð Bjarmason (21):
git-compat-util.h: clarify GCC v.s. C99-specific in comment
C99 support: hard-depend on C99 variadic macros
usage.c: add a die_message() routine
usage.c API users: use die_message() where appropriate
usage.c + gc: add and use a die_message_errno()
config API: don't use vreportf(), make it static in usage.c
common-main.c: call exit(), don't return
usage.c: add a non-fatal bug() function to go with BUG()
parse-options.[ch] API: use bug() to improve error output
receive-pack: use bug() and BUG_if_bug()
cache-tree.c: use bug() and BUG_if_bug()
pack-objects: use BUG(...) not die("BUG: ...")
strbuf.h: use BUG(...) not die("BUG: ...")
usage API: create a new usage.h, move API docs there
usage.[ch] API users: use report_fn, not hardcoded prototype
usage.[ch] API: rename "warn" vars functions to "warning"
usage.c: move usage routines around
usage.c: move rename variables in usage routines around
usage API: use C99 macros for {usage,usagef,die,error,warning,die}*()
usage API: make the "{usage,fatal,error,warning,BUG}: " translatable
usage API: add "core.usageAddSource" config to add <file>:<line>
Documentation/CodingGuidelines | 3 +
Documentation/config/core.txt | 7 +
.../technical/api-error-handling.txt | 81 ------
Documentation/technical/api-trace2.txt | 4 +-
apply.c | 8 +-
apply.h | 6 +-
banned.h | 5 -
builtin/fast-import.c | 22 +-
builtin/gc.c | 21 +-
builtin/notes.c | 15 +-
builtin/pack-objects.c | 2 +-
builtin/receive-pack.c | 16 +-
cache-tree.c | 7 +-
common-main.c | 9 +-
config.c | 22 +-
config.h | 10 +-
daemon.c | 3 +-
git-compat-util.h | 59 +---
http-backend.c | 6 +-
imap-send.c | 3 +-
parse-options.c | 56 ++--
repo-settings.c | 11 +
repository.h | 2 +
run-command.c | 32 +--
strbuf.h | 2 +-
t/helper/test-trace2.c | 27 +-
t/t0210-trace2-normal.sh | 52 ++++
trace.c | 80 +-----
trace.h | 133 ++++-----
trace2.c | 45 +--
trace2.h | 28 --
usage.c | 270 +++++++++++-------
usage.h | 180 ++++++++++++
33 files changed, 636 insertions(+), 591 deletions(-)
delete mode 100644 Documentation/technical/api-error-handling.txt
create mode 100644 usage.h
--
2.34.0.rc2.809.g11e21d44b24
Change a comment added in e208f9cc757 (make error()'s constant return
value more visible, 2012-12-15). It's not correct that this is GCC-ism
anymore, it's code that uses standard C99 features.
The comment being changed here pre-dates the HAVE_VARIADIC_MACROS
define, which we got in e05bed960d3 (trace: add 'file:line' to all
trace output, 2014-07-12).
The original implementation of an error() macro) in e208f9cc757 used a
GCC-ism with the paste operator (see the commit message for mention of
it), but that was dropped later by 9798f7e5f9 (Use __VA_ARGS__ for all
of error's arguments, 2013-02-08), giving us the C99-portable version
we have now.
While we could remove the __GNUC__ define here, it might cause issues
for other compilers or static analysis systems, so let's not. See
87fe5df365 (inline constant return from error() function, 2014-05-06)
for one such issue.
See also e05bed960d3 (trace: add 'file:line' to all trace output,
2014-07-12) for another comment about GNUC's handling of __VA_ARGS__.
Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---
git-compat-util.h | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
We have code in various places that would like to call die(), but
wants to defer the exit(128) it would invoke, e.g. to print an
additional message, or adjust the exit code. Add a die_message()
helper routine to bridge this gap in the API.
Functionally this behaves just like the error() routine, except it'll
print a "fatal: " prefix, and it will exit with 128 instead of -1,
this is so that caller can pas the return value to exit(128), instead
of having to hardcode "128".
A subsequent commit will migrate various callers that benefit from
this function over to it, for now we're just adding the routine and
making die_builtin() in usage.c itself use it.
Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---
git-compat-util.h | 1 +
usage.c | 22 ++++++++++++++++++----
2 files changed, 19 insertions(+), 4 deletions(-)
Remove the "else" branches of the HAVE_VARIADIC_MACROS macro, which
has been unconditionally set since 765dc168882 (git-compat-util:
always enable variadic macros, 2021-01-28).
That commit went out with v2.31.0, so we've got a few releases already
where we've been explicitly breaking things for compilers that don't
support this C99 feature.
In addition to that we've been emitting extremely verbose warnings
since at least ee4512ed481 (trace2: create new combined trace
facility, 2019-02-22) if HAVE_VARIADIC_MACROS wasn't true. There is no
such thing as a "region_enter_printf" or "region_leave_printf" format,
so at least under GCC and Clang everything that includes
trace.h (almost every file) emits a couple of warnings about that.
There's a large benefit to being able to rely on variadic macros, the
code surrounding usage.c is hard to maintain if we need to write two
implementations of everything, and by relying on "__FILE__" and
"__LINE__" along with "__VA_ARGS__" we can in the future make error(),
die() etc. log where they were called from, without the verbosity of a
dual-implementation.
Let's also update our CodingGuidelines to note that we depend on
this. The added bullet-point starts with lower-case for consistency
with other bullet-points in that section.
The diff in "trace.h" is relatively hard to read, since we need to
retain the existing API docs, which were comments on the code used if
HAVE_VARIADIC_MACROS was not defined.
Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---
Documentation/CodingGuidelines | 3 +
banned.h | 5 --
git-compat-util.h | 12 ---
trace.c | 80 +-------------------
trace.h | 133 +++++++++++++--------------------
trace2.c | 39 ----------
trace2.h | 25 -------
usage.c | 15 +---
8 files changed, 58 insertions(+), 254 deletions(-)
@@ -210,6 +210,9 @@ For C programs: . since mid 2017 with 512f41cf, we have been using designated initializers for array (e.g. "int array[10] = { [5] = 2 }").+ . since late 2021 with 765dc168882, we have been using variadic+ macros, mostly for printf-like trace and debug macros.+ These used to be forbidden, but we have not heard any breakage report, and they are assumed to be safe.
@@ -1193,24 +1193,12 @@ static inline int regexec_buf(const regex_t *preg, const char *buf, size_t size,#endif#endif-/*-*Thisisalwaysdefinedasafirststeptowardsmakingtheuseofvariadic-*macrosunconditional.Ifitcausescompilationproblemsonyourplatform,-*pleasereportittotheGitmailinglistatgit@vger.kernel.org.-*/-#define HAVE_VARIADIC_MACROS 1-/* usage.c: only to be used for testing BUG() implementation (see test-tool) */externintBUG_exit_code;-#ifdef HAVE_VARIADIC_MACROS__attribute__((format(printf,3,4)))NORETURNvoidBUG_fl(constchar*file,intline,constchar*fmt,...);#define BUG(...) BUG_fl(__FILE__, __LINE__, __VA_ARGS__)-#else-__attribute__((format(printf,1,2)))NORETURN-voidBUG(constchar*fmt,...);-#endif/**Preserveserrno,printsamessage,butgivesnowarningforENOENT.
Change code that either called error() and proceeded to exit with 128,
or emitted its own "fatal: " messages to use the die_message()
function added in a preceding commit.
In order to do that we need to add a get_die_message_routine()
function, which works like the other get_*_routine() functions in
usage.c. There is no set_die_message_rotine(), as it hasn't been
needed yet. We can add it if we ever need it.
Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---
builtin/fast-import.c | 12 +++++++-----
builtin/notes.c | 9 +++++----
git-compat-util.h | 1 +
http-backend.c | 3 ++-
parse-options.c | 2 +-
run-command.c | 16 +++++-----------
usage.c | 12 ++++++++++--
7 files changed, 31 insertions(+), 24 deletions(-)
@@ -201,11 +201,12 @@ static void prepare_note_data(const struct object_id *object, struct note_data *staticvoidwrite_note_data(structnote_data*d,structobject_id*oid){if(write_object_file(d->buf.buf,d->buf.len,blob_type,oid)){-error(_("unable to write note object"));+intstatus=die_message(_("unable to write note object"));+if(d->edit_path)-error(_("the note contents have been left in %s"),-d->edit_path);-exit(128);+die_message(_("the note contents have been left in %s"),+d->edit_path);+exit(status);}}
@@ -340,15 +340,6 @@ static void child_close_pair(int fd[2])child_close(fd[1]);}-/*-*parentwillmakeitlooklikethechildspewedafatalerroranddied-*thisisneededtopreventchangestot0061.-*/-staticvoidfake_fatal(constchar*err,va_listparams)-{-vreportf("fatal: ",err,params);-}-staticvoidchild_error_fn(constchar*err,va_listparams){constcharmsg[]="error() should not be called in child\n";
Change code the "error: " output when we exit with 128 due to gc.log
errors to use a "fatal: " prefix instead. This adds a sibling function
to the die_errno() added in a preceding commit.
Since it returns 128 instead of -1 we'll need to adjust
report_last_gc_error(). Let's adjust it while we're at it to not
conflate the "should skip" and "exit with this non-zero code"
conditions, as the caller is no longer hardcoding "128", but relying
on die_errno() to return a nen-zero exit() status.
Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---
builtin/gc.c | 21 ++++++++++++---------
git-compat-util.h | 1 +
usage.c | 12 ++++++++++++
3 files changed, 25 insertions(+), 9 deletions(-)
@@ -507,7 +508,7 @@ static int report_last_gc_error(void)"until the file is removed.\n\n""%s"),gc_log_path,sb.buf);-ret=1;+*skip=1;}strbuf_release(&sb);done:
@@ -610,13 +611,15 @@ int cmd_gc(int argc, const char **argv, const char *prefix)fprintf(stderr,_("See \"git help gc\" for manual housekeeping.\n"));}if(detach_auto){-intret=report_last_gc_error();-if(ret<0)-/* an I/O error occurred, already reported */-exit(128);-if(ret==1)+intskip;+intret=report_last_gc_error(&skip);++if(skip)/* Last gc --auto failed. Skip this one. */return0;+if(ret)+/* an error occurred, already reported */+exit(ret);if(lock_repo_for_gc(force,&pid))return0;
Refactor the main() function so that we always take the same path
towards trace2_cmd_exit() whether exit() is invoked, or we end up in
the "return" in the pre-image. This contains no functional change, and
is only intended for the benefit of readers of the code, who'll now be
pointed to our exit() wrapper.
Since ee4512ed481 (trace2: create new combined trace facility,
2019-02-22) we've defined "exit" with a macro to call
trace2_cmd_exit() for us in "git-compat-util.h". So in cases where an
exit() is invoked (such as in several places in "git.c") we don't
reach the trace2_cmd_exit() in the pre-image. This makes it so that
we'll always take that same exit() path.
Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---
common-main.c | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
In preceding commits the rest of the vreportf() users outside of
usage.c have been migrated to die_message(), leaving only the
git_die_config() function added in 5a80e97c827 (config: add
`git_die_config()` to the config-set API, 2014-08-07).
Let's have its callers call error() themselves if they want to emit a
message, which is exactly what git_die_config() was doing for them
before emitting its own die() message.
This means that we can make the vreportf() in usage.c "static", and
only expose functions such as usage(), die(), warning() etc.
Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---
builtin/fast-import.c | 7 ++++---
builtin/notes.c | 6 ++++--
config.c | 22 +++++++++-------------
config.h | 10 ++++++----
git-compat-util.h | 1 -
imap-send.c | 3 ++-
usage.c | 2 +-
7 files changed, 26 insertions(+), 25 deletions(-)
@@ -1350,7 +1350,8 @@ static int git_imap_config(const char *var, const char *val, void *cb)server.port=git_config_int(var,val);elseif(!strcmp("imap.host",var)){if(!val){-git_die_config("imap.host","Missing value for 'imap.host'");+error("Missing value for 'imap.host'");+git_die_config("imap.host");}else{if(starts_with(val,"imap:"))val+=5;
Add a bug() function to use in cases where we'd like to indicate a
runtime BUG(), but would like to deref the BUG() call because we're
possibly accumulating more bug() callers to exhaustively indicate what
went wrong. This will be used in place of optbug() in parse-options.c
Any caller to bug() must follow up such calls with BUG_if_bug(), and
as the tests and documentation here show we'll catch missing
BUG_if_bug() invocations in our exit() wrapper.
I'd previously proposed this as part of another series[1], in that
use-case we ended thinking a BUG() would be better (and eventually
96e41f58fe1 (fsck: report invalid object type-path combinations,
2021-10-01) ended up with neither). Here I'll be converting existing
in-tree users to this, so hopefully this won't be controversial.
I'm not bothering to support the "else" branch of
"HAVE_VARIADIC_MACROS". Since 765dc168882 (git-compat-util: always
enable variadic macros, 2021-01-28) we've had a hard dependency on
them, and manually undefining the macro will nowadays result in a hard
compilation error. We should follow up with [2] instead and remove the
"else" codepath.
1. https://lore.kernel.org/git/YGRXomWwRYPdXZi3@coredump.intra.peff.net/
2. https://lore.kernel.org/git/cover-0.2-00000000000-20210412T105422Z-avarab@gmail.com/
Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---
.../technical/api-error-handling.txt | 13 ++++-
Documentation/technical/api-trace2.txt | 4 +-
git-compat-util.h | 12 +++++
t/helper/test-trace2.c | 27 ++++++++--
t/t0210-trace2-normal.sh | 52 +++++++++++++++++++
trace2.c | 6 +++
usage.c | 32 ++++++++++--
7 files changed, 133 insertions(+), 13 deletions(-)
@@ -1,7 +1,7 @@ Error reporting in git ======================-`BUG`, `die`, `usage`, `error`, and `warning` report errors of+`BUG`, `bug`, `die`, `usage`, `error`, and `warning` report errors of various kinds. - `BUG` is for failed internal assertions that should never happen,
@@ -18,6 +18,17 @@ various kinds. to the user and returns -1 for convenience in signaling the error to the caller.+- `bug` (lower-case, not `BUG`) is supposed to be used like `BUG` but+ returns -1 like error. The user should then call `BUG_if_bug()` to die.+++This is for the convenience of APIs who'd like to potentially report+more than one bug before calling `BUG_if_bug()`, which will invoke+`BUG()` if there were any preceding calls to `bug()`.+++We call `BUG_if_bug()` ourselves in on `exit()` (via a wrapper, not+`atexit()`), which guarantees that we'll catch cases where we forgot+to invoke `BUG_if_bug()` following a call or calls to `bug()`.+ - `warning` is for reporting situations that probably should not occur but which the user (and Git) can continue to work around without running into too many problems. Like `error`, it
@@ -465,8 +465,8 @@ completed.) ------------ `"error"`::- This event is emitted when one of the `BUG()`, `error()`, `die()`,- `warning()`, or `usage()` functions are called.+ This event is emitted when one of the `BUG()`, `bug()`, `error()`,+ `die()`, `warning()`, or `usage()` functions are called. + ------------ {
@@ -1198,9 +1198,21 @@ static inline int regexec_buf(const regex_t *preg, const char *buf, size_t size,/* usage.c: only to be used for testing BUG() implementation (see test-tool) */externintBUG_exit_code;+/* usage.c: if bug() is called we must have a BUG() invocation afterwards */+externintbug_called_must_BUG;+__attribute__((format(printf,3,4)))NORETURNvoidBUG_fl(constchar*file,intline,constchar*fmt,...);#define BUG(...) BUG_fl(__FILE__, __LINE__, __VA_ARGS__)+__attribute__((format(printf,3,4)))+intbug_fl(constchar*file,intline,constchar*fmt,...);+#define bug(...) bug_fl(__FILE__, __LINE__, __VA_ARGS__)+#define BUG_if_bug() do { \+if(bug_called_must_BUG){\+bug_called_must_BUG=0;\+BUG_fl(__FILE__,__LINE__,"see bug() output above");\+}\+}while(0)/**Preserveserrno,printsamessage,butgivesnowarningforENOENT.
@@ -168,6 +168,58 @@ test_expect_success 'BUG messages are written to trace2' 'test_cmpexpectactual'+test_expect_success'bug messages with BUG_if_bug() are written to trace2''+test_when_finished"rm trace.normal actual expect"&&+test_expect_code99envGIT_TRACE2="$(pwd)/trace.normal"\+test-tooltrace2008bug2>err&&+cat>expect<<-\EOF&&+abugmessage+anotherbugmessage+seebug()outputabove+EOF+sed"s/^.*: //"<err>actual&&+test_cmpexpectactual&&++perl"$TEST_DIRECTORY/t0210/scrub_normal.perl"<trace.normal>actual&&+cat>expect<<-EOF&&+version$V+start_EXE_trace2008bug+cmd_nametrace2(trace2)+errorabugmessage+erroranotherbugmessage+errorseebug()outputabove+exitelapsed:_TIME_code:99+atexitelapsed:_TIME_code:99+EOF+test_cmpexpectactual+'++test_expect_success'bug messages without BUG_if_bug() are written to trace2''+test_when_finished"rm trace.normal actual expect"&&+test_expect_code99envGIT_TRACE2="$(pwd)/trace.normal"\+test-tooltrace2009bug_BUG2>err&&+cat>expect<<-\EOF&&+abugmessage+anotherbugmessage+hadbug()outputabove,inadditionmissedBUG_if_bug()call+EOF+sed"s/^.*: //"<err>actual&&+test_cmpexpectactual&&++perl"$TEST_DIRECTORY/t0210/scrub_normal.perl"<trace.normal>actual&&+cat>expect<<-EOF&&+version$V+start_EXE_trace2009bug_BUG+cmd_nametrace2(trace2)+errorabugmessage+erroranotherbugmessage+errorhadbug()outputabove,inadditionmissedBUG_if_bug()call+exitelapsed:_TIME_code:99+atexitelapsed:_TIME_code:99+EOF+test_cmpexpectactual+'+ sane_unsetGIT_TRACE2_BRIEF# Now test without environment variables and get all Trace2 settings
@@ -211,6 +211,12 @@ int trace2_cmd_exit_fl(const char *file, int line, int code)code&=0xff;+if(bug_called_must_BUG){+/* BUG_vfl() calls exit(), which calls us again */+bug_called_must_BUG=0;+BUG("had bug() output above, in addition missed BUG_if_bug() call");+}+if(!trace2_enabled)returncode;
@@ -290,18 +290,24 @@ void warning(const char *warn, ...)/* Only set this, ever, from t/helper/, when verifying that bugs are caught. */intBUG_exit_code;-staticNORETURNvoidBUG_vfl(constchar*file,intline,constchar*fmt,va_listparams)+staticvoidBUG_vfl_common(constchar*file,intline,constchar*fmt,+va_listparams){charprefix[256];-va_listparams_copy;-staticintin_bug;--va_copy(params_copy,params);/* truncation via snprintf is OK here */snprintf(prefix,sizeof(prefix),"BUG: %s:%d: ",file,line);vreportf(prefix,fmt,params);+}++staticNORETURNvoidBUG_vfl(constchar*file,intline,constchar*fmt,va_listparams)+{+va_listparams_copy;+staticintin_bug;++va_copy(params_copy,params);+BUG_vfl_common(file,line,fmt,params);if(in_bug)abort();
Amend code added in a6a84319686 (receive-pack.c: shorten the
execute_commands loop over all commands, 2015-01-07) and amended to
hard die in b6a4788586d (receive-pack.c: die instead of error in case
of possible future bug, 2015-01-07) to the new bug() function instead.
Let's also rename the warn_if_*() function that code is in to
BUG_if_*(), its name became outdated in b6a4788586d.
Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---
builtin/receive-pack.c | 16 ++++++----------
1 file changed, 6 insertions(+), 10 deletions(-)
@@ -1824,21 +1824,17 @@ static int should_process_cmd(struct command *cmd)return!cmd->error_string&&!cmd->skip_update;}-staticvoidwarn_if_skipped_connectivity_check(structcommand*commands,+staticvoidBUG_if_skipped_connectivity_check(structcommand*commands,structshallow_info*si){structcommand*cmd;-intchecked_connectivity=1;for(cmd=commands;cmd;cmd=cmd->next){-if(should_process_cmd(cmd)&&si->shallow_ref[cmd->index]){-error("BUG: connectivity check has not been run on ref %s",-cmd->ref_name);-checked_connectivity=0;-}+if(!should_process_cmd(cmd)&&si->shallow_ref[cmd->index])+bug("connectivity check has not been run on ref %s",+cmd->ref_name);}-if(!checked_connectivity)-BUG("connectivity check skipped???");+BUG_if_bug();}staticvoidexecute_commands_non_atomic(structcommand*commands,
Change "BUG" output originally added in a97e4075a16 (Keep
rename/rename conflicts of intermediate merges while doing recursive
merge, 2007-03-31), and later made to say it was a "BUG" in
19c6a4f8369 (merge-recursive: do not return NULL only to cause
segfault, 2010-01-21) to use the new bug() function.
This gets the same job done with less code, this changes the output a
bit, but since we're emitting BUG output let's say it's OK to prefix
every line with the "unmerged index entry" message, instead of
optimizing for readability. doing it this way gets rid of any state
management in the loop itself in favor of BUG_if_bug().
Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---
cache-tree.c | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
@@ -692,14 +692,13 @@ struct tree* write_in_core_index_as_tree(struct repository *repo) {ret=write_index_as_tree_internal(&o,index_state,was_valid,0,NULL);if(ret==WRITE_TREE_UNMERGED_INDEX){inti;-fprintf(stderr,"BUG: There are unmerged index entries:\n");for(i=0;i<index_state->cache_nr;i++){conststructcache_entry*ce=index_state->cache[i];if(ce_stage(ce))-fprintf(stderr,"BUG: %d %.*s\n",ce_stage(ce),-(int)ce_namelen(ce),ce->name);+bug("unmerged index entry on in-memory index write: %d %.*s",+ce_stage(ce),(int)ce_namelen(ce),ce->name);}-BUG("unmerged index entries when writing inmemory index");+BUG_if_bug();}returnlookup_tree(repo,&index_state->cache_tree->oid);
Change this code added in da93d12b004 (pack-objects: be incredibly
anal about stdio semantics, 2006-04-02) to use BUG() instead.
See 1a07e59c3e2 (Update messages in preparation for i18n, 2018-07-21)
for when the "BUG: " prefix was added, and [1] for background on the
Solaris behavior that prompted the exhaustive error checking in this
fgets() loop.
1. https://lore.kernel.org/git/824.1144007555@lotus.CS.Berkeley.EDU/
Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---
builtin/pack-objects.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
@@ -3397,7 +3397,7 @@ static void read_object_list_from_stdin(void)if(feof(stdin))break;if(!ferror(stdin))-die("BUG: fgets returned NULL, not EOF, not error!");+BUG("fgets returned NULL, not EOF, not error!");if(errno!=EINTR)die_errno("fgets");clearerr(stdin);
A move-only change to re-order the usage routines in the order of
usage,die,die_message,error,warning, and to have the "errno" variant
after the non-errno variant, in addition to defining them all after
the fmt_with_err() helper which some of them need.
This change make a subsequent non-refactoring commit's diff smaller.
Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---
usage.c | 84 ++++++++++++++++++++++++++++-----------------------------
1 file changed, 42 insertions(+), 42 deletions(-)
@@ -154,34 +154,6 @@ void set_die_is_recursing_routine(int (*routine)(void))die_is_recursing=routine;}-voidNORETURNusagef(constchar*err,...)-{-va_listparams;--va_start(params,err);-usage_routine(err,params);-va_end(params);-}--voidNORETURNusage(constchar*err)-{-usagef("%s",err);-}--voidNORETURNdie(constchar*err,...)-{-va_listparams;--if(die_is_recursing()){-fputs("fatal: recursion detected in die handler\n",stderr);-exit(128);-}--va_start(params,err);-die_routine(err,params);-va_end(params);-}-staticconstchar*fmt_with_err(char*buf,intn,constchar*fmt){charstr_error[256],*err;
@@ -206,6 +178,34 @@ static const char *fmt_with_err(char *buf, int n, const char *fmt)returnbuf;}+voidNORETURNusage(constchar*err)+{+usagef("%s",err);+}++voidNORETURNusagef(constchar*err,...)+{+va_listparams;++va_start(params,err);+usage_routine(err,params);+va_end(params);+}++voidNORETURNdie(constchar*err,...)+{+va_listparams;++if(die_is_recursing()){+fputs("fatal: recursion detected in die handler\n",stderr);+exit(128);+}++va_start(params,err);+die_routine(err,params);+va_end(params);+}+voidNORETURNdie_errno(constchar*fmt,...){charbuf[1024];
Optionally extend the support that BUG() has had for emitting line
numbers to the {usage,fatal,error,warning}{,_errno}() functions.
Before we'd unconditionally get error messages like:
$ git -c core.usageAddSource=false -c core.x=y config --get --bool core.x
fatal: bad boolean config value 'y' for 'core.x'
Which can be changed with core.usageAddSource=true to include the file
and line numbers:
$ git -c core.usageAddSource=false -c core.x=y config --get --bool core.x
fatal: config.c:1241: bad boolean config value 'y' for 'core.x'
As the added documentation notes this is primarily intended to be
useful to developers of git itself, but is being exposed as a user
setting to e.g. help file better bug reports.
This also adds a "GIT_TEST_USAGE_ADD_SOURCE" setting intended to run
the test suite in this mode.
Currently it has a lot of failures. Most of those are rather trivial,
and can be "fixed" by pointing GIT_TEST_CMP to a "diff -u" that does a
s/^(usage|fatal|error|warning): [^:]+:[0-9]+/$1/g on its input files,
and likewise for a "grep" wrapper that does the same.
Even if we can't run the tests in this mode yet I'd like to have this
for ad-hoc debugging, and to make it easier to work towards running
the tests in this mode. If we can turn this on permanently it'll be
much easier to read test output, as we won't need to worry about the
indirection of looking up where an error might have been emitted,
which can be especially painful when the message being emitted isn't
unique within git.git.
This new code needs to be guarded by the "dying" variable for the
reasons explained in 2d3c02f5db6 (die(): stop hiding errors due to
overzealous recursion guard, 2017-06-21), and for those same reasons
it's racy under multi-threading.
Here the worst case is that incrementing the variable will run away
from us, and we won't get our desired <file>:<line> number. That's OK
in this case, those cases should be isolated to the config code (if we
can't parse the config), memory allocation etc, but we'll get it right
in the common cases.
Using GIT_TEST_USAGE_ADD_SOURCE should be immune from any racyness, as
it only needs a getenv() and git_parse_maybe_bool(), which won't die.
Add a repo_cfg_bool_env() wrapper to repo-settings.c for
GIT_TEST_USAGE_ADD_SOURCE, in 3050b6dfc75 (repo-settings.c: simplify
the setup, 2021-09-21) I indicated that the GIT_TEST_MULTI_PACK_INDEX
env variable/config pair in that file has odd semantics, but users of
repo_cfg_bool_env() have straightforward and expected semantics. If
the environment variable is set (true or false) we'll use it,
otherwise we'll use the config, and finally fall back on the
default (of "false", in this case).
Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---
Documentation/config/core.txt | 7 +++++++
repo-settings.c | 11 +++++++++++
repository.h | 2 ++
usage.c | 24 ++++++++++++++++++++++--
4 files changed, 42 insertions(+), 2 deletions(-)
@@ -101,6 +101,13 @@ core.untrackedCache:: `feature.manyFiles` is enabled which sets this setting to `true` by default.+core.usageAddSource::+ Adds the source "<file>:<line>" numbers to usage messages+ ("usage:", "fatal:", "error:", "warning:"). This setting is+ primarily intended for those debugging or developing git+ itself, or to compose a bug report against git itself (see+ linkgit:git-bugreport[1]).+ core.checkStat:: When missing or is set to `default`, many fields in the stat structure are checked to detect if a file has been modified
@@ -43,7 +64,7 @@ static void vreportf(enum usage_kind kind,}/* truncation via snprintf is OK here */-if(kind==USAGE_BUG)+if(kind==USAGE_BUG||addln)snprintf(prefix,sizeof(prefix),"%s%s:%d: ",prefix_i18n,file,line);elsesnprintf(prefix,sizeof(prefix),"%s",prefix_i18n);
@@ -123,7 +144,6 @@ static void warning_builtin(const char *file, int line, const char *warn, va_lisstaticintdie_is_recursing_builtin(void){-staticintdying;/**JustanarbitrarynumberXwhere"a < x < b"where"a"is*"maximum number of pthreads we'll ever plausibly spawn"and
In preceding commits the vreportf() function was made static, so we
know it's only being called with a limited set of fixed prefixes. Pass
an enum indicating the kind of usage message we're emitting instead,
which means that we can fold the BUG_vfl_common() functionality
directly into it.
Since we've now got one place were we're emitting these usage messages
we can make them translatable.
We need to be careful with this function to not malloc() anything, as
a failure in say a use of strbuf_vaddf() would call xmalloc(), which
would in turn call die(), but here we're using static strings, either
from libintl or not.
I was on the fence about making the "BUG: " message translatable, but
let's do it for consistency. Someone who doesn't speak a word of
English may not know what "BUG" means, but if it's translated they
might have an easier time knowing that they have to report a bug
upstream. Since we'll always emit the line number it's unlikely that
we're going to be confused by such a report.
As we've moved the BUG_vfl_common() code into vsnprintf() we can do
away with one of the two checks for buffer sizes added in
116d1fa6c69 (vreportf(): avoid relying on stdio buffering, 2019-10-30)
and ac4896f007a (fmt_with_err: add a comment that truncation is OK,
2018-05-18).
I.e. we're being overly paranoid if we define the fixed-size "prefix"
and "msg" buffers, are OK with the former being truncated, and then
effectively check if our 256-byte buffer is larger than our 4096-byte
buffer. I wondered about adding a:
assert(sizeof(prefix) < sizeof(msg)); /* overly paranoid much? */
But I think that would be overdoing it. Anyone modifying this function
will keep these two buffer sizes in mind, so let's just remove one of
the checks instead.
Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---
usage.c | 68 ++++++++++++++++++++++++++++++++++++++-------------------
1 file changed, 45 insertions(+), 23 deletions(-)
@@ -6,16 +6,49 @@#include"git-compat-util.h"#include"cache.h"-staticvoidvreportf(constchar*prefix,constchar*err,va_listparams)+enumusage_kind{+USAGE_USAGE,+USAGE_DIE,+USAGE_ERROR,+USAGE_WARNING,+USAGE_BUG,+};++staticvoidvreportf(enumusage_kindkind,+constchar*file,intline,+constchar*err,va_listparams){+constchar*prefix_i18n;+charprefix[256];charmsg[4096];char*p,*pend=msg+sizeof(msg);-size_tprefix_len=strlen(prefix);--if(sizeof(msg)<=prefix_len){-fprintf(stderr,"BUG!!! too long a prefix '%s'\n",prefix);-abort();+size_tprefix_len;++switch(kind){+caseUSAGE_USAGE:+prefix_i18n=_("usage: ");+break;+caseUSAGE_DIE:+prefix_i18n=_("fatal: ");+break;+caseUSAGE_ERROR:+prefix_i18n=_("error: ");+break;+caseUSAGE_WARNING:+prefix_i18n=_("warning: ");+break;+caseUSAGE_BUG:+prefix_i18n=_("BUG: ");+break;}++/* truncation via snprintf is OK here */+if(kind==USAGE_BUG)+snprintf(prefix,sizeof(prefix),"%s%s:%d: ",prefix_i18n,file,line);+else+snprintf(prefix,sizeof(prefix),"%s",prefix_i18n);++prefix_len=strlen(prefix);memcpy(msg,prefix,prefix_len);p=msg+prefix_len;if(vsnprintf(p,pend-p,err,params)<0)
@@ -283,24 +316,13 @@ void warning_errno_fl(const char *file, int line, const char *fmt, ...)/* Only set this, ever, from t/helper/, when verifying that bugs are caught. */intBUG_exit_code;-staticvoidBUG_vfl_common(constchar*file,intline,constchar*fmt,-va_listparams)-{-charprefix[256];--/* truncation via snprintf is OK here */-snprintf(prefix,sizeof(prefix),"BUG: %s:%d: ",file,line);--vreportf(prefix,fmt,params);-}-staticNORETURNvoidBUG_vfl(constchar*file,intline,constchar*fmt,va_listparams){va_listparams_copy;staticintin_bug;va_copy(params_copy,params);-BUG_vfl_common(file,line,fmt,params);+vreportf(USAGE_BUG,file,line,fmt,params);if(in_bug)abort();
@@ -330,7 +352,7 @@ int bug_fl(const char *file, int line, const char *fmt, ...)va_copy(cp,ap);va_start(ap,fmt);-BUG_vfl_common(file,line,fmt,ap);+vreportf(USAGE_BUG,file,line,fmt,ap);va_end(ap);trace2_cmd_error_va_fl(file,line,fmt,cp);
Change the "usage" family of functions to be defined in terms of C99
variadic macros, as we've optionally done with the BUG() macro and
BUG_fl() function since d8193743e08 (usage.c: add BUG() function,
2017-05-12), and unconditionally since 765dc168882 (git-compat-util:
always enable variadic macros, 2021-01-28).
This would have been possible before having a hard dependency on C99,
but as the dual implementations of C99 and pre-C99 macros and
functions adjusted in preceding commits show, doing so would have been
rather painful.
By having these be macros we'll now log meaningful "file" and "line"
entries in trace2 events. Before this we'd log "usage.c" in all of
them, and the line would be the relevant locations in that file.
To do this we need to not only introduce X_fl() functions for
{die,error,warning,die}{,_errno}(), but also change all the callers of
the set_*() and get_() functions in usage.h to take "file" and "line"
arguments.
Neither the built-in {die,error,warning,die}{,_errno}() nor anyone
else does anything useful with these "file" and "line" arguments for
now, but it means we can define all our macros and functions
consistently.
It also opens the door for a follow-up change where these functions
could optionally emit the file name and line number, e.g. for
DEVELOPER=1 builds, or depending on configuration.
It might be a good change to remove the "fmt" key from the "error"
events as a follow-up change. As these few examples from running the
test suite show it's sometimes redundant (same as the "msg"), rather
useless (just a "%s"), or something we could now mostly aggregate by
file/line instead of the normalized printf format:
1 file":"builtin/gc.c","line":1391,"msg":"'bogus' is not a valid task","fmt":"'%s' is not a valid task"}
1 file":"builtin/for-each-ref.c","line":89,"msg":"format: %(then) atom used more than once","fmt":"%s"}
1 file":"builtin/fast-import.c","line":411,"msg":"Garbage after mark: N :202 :302x","fmt":"Garbage after mark: %s"}
"Mostly" here assumes that it would be OK if the aggregation changed
between git versions, which may be what all users of trace2 want. The
change that introduced the "fmt" code was ee4512ed481 (trace2: create
new combined trace facility, 2019-02-22), and the documentation change
was e544221d97a (trace2: Documentation/technical/api-trace2.txt,
2019-02-22).
Both are rather vague on what problem "fmt" solved exactly, aside from
the obvious one of it being impossible to do meaningful aggregations
due to the "file" and "line" being the same everywhere, which isn't
the case now.
In any case, let's leave "fmt" be for now, the above summary was given
in case it's interesting to remove it in the future, e.g. to save
space in trace2 payloads.
Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---
apply.c | 2 +-
builtin/fast-import.c | 5 +--
daemon.c | 3 +-
http-backend.c | 5 +--
run-command.c | 15 ++++----
trace2.h | 3 --
usage.c | 79 ++++++++++++++++++++-----------------------
usage.h | 49 +++++++++++++++++++--------
8 files changed, 87 insertions(+), 74 deletions(-)
@@ -123,7 +123,7 @@ void clear_apply_state(struct apply_state *state)/* &state->fn_table is cleared at the end of apply_patch() */}-staticvoidmute_routine(constchar*msg,va_listparams)+staticvoidmute_routine(constchar*file,intline,constchar*msg,va_listparams){/* do nothing */}
@@ -340,19 +340,19 @@ static void child_close_pair(int fd[2])child_close(fd[1]);}-staticvoidchild_error_fn(constchar*err,va_listparams)+staticvoidchild_error_fn(constchar*file,intline,constchar*err,va_listparams){constcharmsg[]="error() should not be called in child\n";xwrite(2,msg,sizeof(msg)-1);}-staticvoidchild_warning_fn(constchar*err,va_listparams)+staticvoidchild_warn_fn(constchar*file,intline,constchar*err,va_listparams){-constcharmsg[]="warning() should not be called in child\n";+constcharmsg[]="warn() should not be called in child\n";xwrite(2,msg,sizeof(msg)-1);}-staticvoidNORETURNchild_die_fn(constchar*err,va_listparams)+staticvoidNORETURNchild_die_fn(constchar*file,intline,constchar*err,va_listparams){constcharmsg[]="die() should not be called in child\n";xwrite(2,msg,sizeof(msg)-1);
@@ -783,7 +783,7 @@ int start_command(struct child_process *cmd)*/set_die_routine(child_die_fn);set_error_routine(child_error_fn);-set_warning_routine(child_warning_fn);+set_warning_routine(child_warn_fn);close(notify_pipe[0]);set_cloexec(notify_pipe[1]);
@@ -120,9 +120,6 @@ int trace2_cmd_exit_fl(const char *file, int line, int code);voidtrace2_cmd_error_va_fl(constchar*file,intline,constchar*fmt,va_listap);-#define trace2_cmd_error_va(fmt, ap) \-trace2_cmd_error_va_fl(__FILE__,__LINE__,(fmt),(ap))-/**Emita'pathname'eventwiththecanonicalpathnameofthecurrentprocess*Thisgivespost-processorsasimplefieldtoidentifythecommandwithout
A renaming-only change to rename variables in the usage routines to be
consistent. Before we'd use "params", now we use "ap", and the mixture
of "fmt", "err", "warn" etc. is replaced with just "fmt".
This change make a subsequent non-refactoring commit's diff smaller
and easier to understand.
Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---
usage.c | 88 ++++++++++++++++++++++++++++-----------------------------
1 file changed, 44 insertions(+), 44 deletions(-)
@@ -178,38 +178,38 @@ static const char *fmt_with_err(char *buf, int n, const char *fmt)returnbuf;}-voidNORETURNusage(constchar*err)+voidNORETURNusage(constchar*fmt){-usagef("%s",err);+usagef("%s",fmt);}-voidNORETURNusagef(constchar*err,...)+voidNORETURNusagef(constchar*fmt,...){-va_listparams;+va_listap;-va_start(params,err);-usage_routine(err,params);-va_end(params);+va_start(ap,fmt);+usage_routine(fmt,ap);+va_end(ap);}-voidNORETURNdie(constchar*err,...)+voidNORETURNdie(constchar*fmt,...){-va_listparams;+va_listap;if(die_is_recursing()){fputs("fatal: recursion detected in die handler\n",stderr);exit(128);}-va_start(params,err);-die_routine(err,params);-va_end(params);+va_start(ap,fmt);+die_routine(fmt,ap);+va_end(ap);}voidNORETURNdie_errno(constchar*fmt,...){charbuf[1024];-va_listparams;+va_listap;if(die_is_recursing()){fputs("fatal: recursion detected in die_errno handler\n",
@@ -237,22 +237,22 @@ int die_message(const char *err, ...)intdie_message_errno(constchar*fmt,...){charbuf[1024];-va_listparams;+va_listap;-va_start(params,fmt);-die_message_routine(fmt_with_err(buf,sizeof(buf),fmt),params);-va_end(params);+va_start(ap,fmt);+die_message_routine(fmt_with_err(buf,sizeof(buf),fmt),ap);+va_end(ap);return-1;}#undef error-interror(constchar*err,...)+interror(constchar*fmt,...){-va_listparams;+va_listap;-va_start(params,err);-error_routine(err,params);-va_end(params);+va_start(ap,fmt);+error_routine(fmt,ap);+va_end(ap);return-1;}
@@ -260,31 +260,31 @@ int error(const char *err, ...)interror_errno(constchar*fmt,...){charbuf[1024];-va_listparams;+va_listap;-va_start(params,fmt);-error_routine(fmt_with_err(buf,sizeof(buf),fmt),params);-va_end(params);+va_start(ap,fmt);+error_routine(fmt_with_err(buf,sizeof(buf),fmt),ap);+va_end(ap);return-1;}-voidwarning(constchar*warn,...)+voidwarning(constchar*fmt,...){-va_listparams;+va_listap;-va_start(params,warn);-warning_routine(warn,params);-va_end(params);+va_start(ap,fmt);+warning_routine(fmt,ap);+va_end(ap);}-voidwarning_errno(constchar*warn,...)+voidwarning_errno(constchar*fmt,...){charbuf[1024];-va_listparams;+va_listap;-va_start(params,warn);-warning_routine(fmt_with_err(buf,sizeof(buf),warn),params);-va_end(params);+va_start(ap,fmt);+warning_routine(fmt_with_err(buf,sizeof(buf),fmt),ap);+va_end(ap);}/* Only set this, ever, from t/helper/, when verifying that bugs are caught. */
There is a warning() routine, not warn(), but parts of the function
interface confusingly used "warn". Let's rename these for consistency.
Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---
apply.c | 6 +++---
apply.h | 4 ++--
run-command.c | 8 ++++----
usage.c | 16 ++++++++--------
usage.h | 4 ++--
5 files changed, 19 insertions(+), 19 deletions(-)
@@ -160,9 +160,9 @@ int check_apply_state(struct apply_state *state, int force_apply)if(state->apply_verbosity<=verbosity_silent){state->saved_error_routine=get_error_routine();-state->saved_warn_routine=get_warn_routine();+state->saved_warning_routine=get_warning_routine();set_error_routine(mute_routine);-set_warn_routine(mute_routine);+set_warning_routine(mute_routine);}return0;
@@ -4999,7 +4999,7 @@ int apply_all_patches(struct apply_state *state,if(state->apply_verbosity<=verbosity_silent){set_error_routine(state->saved_error_routine);-set_warn_routine(state->saved_warn_routine);+set_warning_routine(state->saved_warning_routine);}if(res>-1)
@@ -346,9 +346,9 @@ static void child_error_fn(const char *err, va_list params)xwrite(2,msg,sizeof(msg)-1);}-staticvoidchild_warn_fn(constchar*err,va_listparams)+staticvoidchild_warning_fn(constchar*err,va_listparams){-constcharmsg[]="warn() should not be called in child\n";+constcharmsg[]="warning() should not be called in child\n";xwrite(2,msg,sizeof(msg)-1);}
@@ -778,12 +778,12 @@ int start_command(struct child_process *cmd)if(!cmd->pid){intsig;/*-*Ensurethedefaultdie/error/warnroutinesdonotget+*Ensurethedefaultdie/error/warningroutinesdonotget*called,theycantakestdiolocksandmalloc.*/set_die_routine(child_die_fn);set_error_routine(child_error_fn);-set_warn_routine(child_warn_fn);+set_warning_routine(child_warning_fn);close(notify_pipe[0]);set_cloexec(notify_pipe[1]);
Follow-up 26c816a67de (Merge branch 'hw/doc-in-header', 2019-12-16)
and move the API docs for the usage.c API to a new usage.h, which is
included from "git-compat-util.h".
All of the code and documentation is unchanged here, with the
following exceptions:
* Added a short paragraph to the start of the comment in usage.h
describing the API in general terms.
* Prefixed comments with " * ", reformatted "+"-flowing lines away
from that ASCIIDOC syntax.
* Re-arranged the function definitions that were previously in
"git-compat-util.h" to be grouped by task, and added short comments
above each one. None of the code was changed.
* There was an unrelated forward-declaration of "struct strbuf" that
happened to be just above these usage functions. Move it down to just
above "unlink_or_msg()" where it's needed.
Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---
.../technical/api-error-handling.txt | 92 ----------
git-compat-util.h | 59 +------
usage.h | 161 ++++++++++++++++++
3 files changed, 163 insertions(+), 149 deletions(-)
delete mode 100644 Documentation/technical/api-error-handling.txt
create mode 100644 usage.h
@@ -1,92 +0,0 @@-Error reporting in git-======================--`BUG`, `bug`, `die`, `usage`, `error`, and `warning` report errors of-various kinds.--- `BUG` is for failed internal assertions that should never happen,- i.e. a bug in git itself.--- `die` is for fatal application errors. It prints a message to- the user and exits with status 128.--- `usage` is for errors in command line usage. After printing its- message, it exits with status 129. (See also `usage_with_options`- in the link:api-parse-options.html[parse-options API].)--- `error` is for non-fatal library errors. It prints a message- to the user and returns -1 for convenience in signaling the error- to the caller.--- `bug` (lower-case, not `BUG`) is supposed to be used like `BUG` but- returns -1 like error. The user should then call `BUG_if_bug()` to die.-+-This is for the convenience of APIs who'd like to potentially report-more than one bug before calling `BUG_if_bug()`, which will invoke-`BUG()` if there were any preceding calls to `bug()`.-+-We call `BUG_if_bug()` ourselves in on `exit()` (via a wrapper, not-`atexit()`), which guarantees that we'll catch cases where we forgot-to invoke `BUG_if_bug()` following a call or calls to `bug()`.--- `warning` is for reporting situations that probably should not- occur but which the user (and Git) can continue to work around- without running into too many problems. Like `error`, it- returns -1 after reporting the situation to the caller.--These reports will be logged via the trace2 facility. See the "error"-event in link:api-trace2.txt[trace2 API].--Customizable error handlers------------------------------The default behavior of `die` and `error` is to write a message to-stderr and then exit or return as appropriate. This behavior can be-overridden using `set_die_routine` and `set_error_routine`. For-example, "git daemon" uses set_die_routine to write the reason `die`-was called to syslog before exiting.--Library errors-----------------Functions return a negative integer on error. Details beyond that-vary from function to function:--- Some functions return -1 for all errors. Others return a more- specific value depending on how the caller might want to react- to the error.--- Some functions report the error to stderr with `error`,- while others leave that for the caller to do.--- errno is not meaningful on return from most functions (except- for thin wrappers for system calls).--Check the function's API documentation to be sure.--Caller-handled errors------------------------An increasing number of functions take a parameter 'struct strbuf *err'.-On error, such functions append a message about what went wrong to the-'err' strbuf. The message is meant to be complete enough to be passed-to `die` or `error` as-is. For example:-- if (ref_transaction_commit(transaction, &err))- die("%s", err.buf);--The 'err' parameter will be untouched if no error occurred, so multiple-function calls can be chained:-- t = ref_transaction_begin(&err);- if (!t ||- ref_transaction_update(t, "HEAD", ..., &err) ||- ret_transaction_commit(t, &err))- die("%s", err.buf);--The 'err' parameter must be a pointer to a valid strbuf. To silence-a message, pass a strbuf that is explicitly ignored:-- if (thing_that_can_fail_in_an_ignorable_way(..., &err))- /* This failure is okay. */- strbuf_reset(&err);
@@ -1195,25 +1158,6 @@ static inline int regexec_buf(const regex_t *preg, const char *buf, size_t size,#endif#endif-/* usage.c: only to be used for testing BUG() implementation (see test-tool) */-externintBUG_exit_code;--/* usage.c: if bug() is called we must have a BUG() invocation afterwards */-externintbug_called_must_BUG;--__attribute__((format(printf,3,4)))NORETURN-voidBUG_fl(constchar*file,intline,constchar*fmt,...);-#define BUG(...) BUG_fl(__FILE__, __LINE__, __VA_ARGS__)-__attribute__((format(printf,3,4)))-intbug_fl(constchar*file,intline,constchar*fmt,...);-#define bug(...) bug_fl(__FILE__, __LINE__, __VA_ARGS__)-#define BUG_if_bug() do { \-if(bug_called_must_BUG){\-bug_called_must_BUG=0;\-BUG_fl(__FILE__,__LINE__,"see bug() output above");\-}\-}while(0)-/**Preserveserrno,printsamessage,butgivesnowarningforENOENT.*Returns0onsuccess,whichincludestryingtounlinkanobjectthatdoes
@@ -1226,6 +1170,7 @@ int unlink_or_warn(const char *path);*appendsamessagetoerrsuitablefor*'error("%s",err->buf)'onerror.*/+structstrbuf;intunlink_or_msg(constchar*file,structstrbuf*err);/**Preserveserrno,printsamessage,butgivesnowarningforENOENT.
@@ -0,0 +1,161 @@+#ifndef USAGE_H+#define USAGE_H++/**+*Theusage.hisanAPIforerrorreportingingit,errorsare+*reportedbothtotheuser,toTrace2(see"trace2.h"),andpossibly+*tocustomcallbacksvia"report_fn"callbacks.+*+*`BUG`,`bug`,`die`,`usage`,`error`,and`warning`reporterrorsof+*variouskinds.+*+*-`BUG`isforfailedinternalassertionsthatshouldneverhappen,+*i.e.abugingititself.+*+*-`die`isforfatalapplicationerrors.Itprintsamessageto+*theuserandexitswithstatus128.+*+*-`usage`isforerrorsincommandlineusage.Afterprintingits+*message,itexitswithstatus129.(Seealso`usage_with_options`+*inthelink:api-parse-options.html[parse-optionsAPI].)+*+*-`error`isfornon-fatallibraryerrors.Itprintsamessage+*totheuserandreturns-1forconvenienceinsignalingtheerror+*tothecaller.+*+*-`bug`(lower-case,not`BUG`)issupposedtobeusedlike`BUG`but+*returns-1likeerror.Theusershouldthencall`BUG_if_bug()`todie.+*+*ThisisfortheconvenienceofAPIswho'dliketopotentiallyreport+*morethanonebugbeforecalling`BUG_if_bug()`,whichwillinvoke+*`BUG()`iftherewereanyprecedingcallsto`bug()`.+*+*Wecall`BUG_if_bug()`ourselvesinon`exit()`(viaawrapper,not+*`atexit()`),whichguaranteesthatwe'llcatchcaseswhereweforgot+*toinvoke`BUG_if_bug()`followingacallorcallsto`bug()`.+*+*-`warning`isforreportingsituationsthatprobablyshouldnot+*occurbutwhichtheuser(andGit)cancontinuetoworkaround+*withoutrunningintotoomanyproblems.Like`error`,it+*returns-1afterreportingthesituationtothecaller.+*+*Thesereportswillbeloggedviathetrace2facility.Seethe"error"+*eventinlink:api-trace2.txt[trace2API].+*+*Customizableerrorhandlers+*---------------------------+*+*Thedefaultbehaviorof`die`and`error`istowriteamessageto+*stderrandthenexitorreturnasappropriate.Thisbehaviorcanbe+*overriddenusing`set_die_routine`and`set_error_routine`.For+*example,"git daemon"usesset_die_routinetowritethereason`die`+*wascalledtosyslogbeforeexiting.+*+*Libraryerrors+*--------------+*+*Functionsreturnanegativeintegeronerror.Detailsbeyondthat+*varyfromfunctiontofunction:+*+*-Somefunctionsreturn-1forallerrors.Othersreturnamore+*specificvaluedependingonhowthecallermightwanttoreact+*totheerror.+*+*-Somefunctionsreporttheerrortostderrwith`error`,+*whileothersleavethatforthecallertodo.+*+*-errnoisnotmeaningfulonreturnfrommostfunctions(except+*forthinwrappersforsystemcalls).+*+*Checkthefunction'sAPIdocumentationtobesure.+*+*Caller-handlederrors+*---------------------+*+*Anincreasingnumberoffunctionstakeaparameter'structstrbuf*err'.+*Onerror,suchfunctionsappendamessageaboutwhatwentwrongtothe+*'err'strbuf.Themessageismeanttobecompleteenoughtobepassed+*to`die`or`error`as-is.Forexample:+*+*if(ref_transaction_commit(transaction,&err))+*die("%s",err.buf);+*+*The'err'parameterwillbeuntouchedifnoerroroccurred,somultiple+*functioncallscanbechained:+*+*t=ref_transaction_begin(&err);+*if(!t||+*ref_transaction_update(t,"HEAD",...,&err)||+*ret_transaction_commit(t,&err))+*die("%s",err.buf);+*+*The'err'parametermustbeapointertoavalidstrbuf.Tosilence+*amessage,passastrbufthatisexplicitlyignored:+*+*if(thing_that_can_fail_in_an_ignorable_way(...,&err))+*// This failure is okay.+*strbuf_reset(&err);+*/++/**+*Externalbutprivatevariables,don'tusetheseexceptfor+*implementationdetailsofthisAPIitself.+*/+/* Only to be used for testing BUG() implementation (see test-tool) */+externintBUG_exit_code;+/* If bug() is called we must have a BUG() invocation afterwards */+externintbug_called_must_BUG;++/* General helper functions */+NORETURNvoidusage(constchar*err);+NORETURNvoidusagef(constchar*err,...)__attribute__((format(printf,1,2)));+NORETURNvoiddie(constchar*err,...)__attribute__((format(printf,1,2)));+NORETURNvoiddie_errno(constchar*err,...)__attribute__((format(printf,1,2)));+intdie_message(constchar*err,...)__attribute__((format(printf,1,2)));+intdie_message_errno(constchar*err,...)__attribute__((format(printf,1,2)));+interror(constchar*err,...)__attribute__((format(printf,1,2)));+interror_errno(constchar*err,...)__attribute__((format(printf,1,2)));+voidwarning(constchar*err,...)__attribute__((format(printf,1,2)));+voidwarning_errno(constchar*err,...)__attribute__((format(printf,1,2)));++/* General helper functions invoked via macro wrappers */+__attribute__((format(printf,3,4)))NORETURN+voidBUG_fl(constchar*file,intline,constchar*fmt,...);+__attribute__((format(printf,3,4)))+intbug_fl(constchar*file,intline,constchar*fmt,...);++/* General helper macros */+#define BUG(...) BUG_fl(__FILE__, __LINE__, __VA_ARGS__)+#define bug(...) bug_fl(__FILE__, __LINE__, __VA_ARGS__)+#define BUG_if_bug() do { \+if(bug_called_must_BUG){\+bug_called_must_BUG=0;\+BUG_fl(__FILE__,__LINE__,"see bug() output above");\+}\+}while(0)++/* Setting custom handling routines */+typedefvoid(*report_fn)(constchar*,va_listparams);+voidset_die_routine(NORETURN_PTRreport_fnroutine);+report_fnget_die_message_routine(void);+voidset_error_routine(report_fnroutine);+report_fnget_error_routine(void);+voidset_warn_routine(report_fnroutine);+report_fnget_warn_routine(void);+voidset_die_is_recursing_routine(int(*routine)(void));++/*+*Letcallersbeawareoftheconstantreturnvalue;thiscanhelp+*gccwith-Wuninitializedanalysis.Werestrictthistricktogcc,though,+*becauseothercompilersmaybeconfusedbythis.+*/+#if defined(__GNUC__)+staticinlineintconst_error(void)+{+return-1;+}+#define error(...) (error(__VA_ARGS__), const_error())+#define error_errno(...) (error_errno(__VA_ARGS__), const_error())+#endif++#endif
Change a couple of users of "report_fn" that hardcoded a definition of
it to use the definition of report_fn instead.
Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---
apply.h | 4 ++--
run-command.c | 3 +--
2 files changed, 3 insertions(+), 4 deletions(-)
@@ -362,10 +362,9 @@ static void NORETURN child_die_fn(const char *err, va_list params)/* this runs in the parent process */staticvoidchild_err_spew(structchild_process*cmd,structchild_err*cerr){-staticvoid(*old_errfn)(constchar*err,va_listparams);+report_fnold_errfn=get_error_routine();report_fndie_message_routine=get_die_message_routine();-old_errfn=get_error_routine();set_error_routine(die_message_routine);errno=cerr->syserr;
In 7141efab248 (strbuf: clarify assertion in strbuf_setlen(),
2011-04-27) this 'die("BUG: "' invocation was added with the rationale
that strbuf.c had existing users doing the same, but those users were
later changed to use BUG() in 033abf97fcb (Replace all die("BUG: ...")
calls by BUG() ones, 2018-05-02). Let's do the same here.
Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---
strbuf.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
When we run into bugs in parse-options.c usage it's good to be able to
note all the issues we ran into before dying, which is why we have the
optbug() function.
Let's instead use the bug() helper function that's newly added to
usage.c to do the same thing, which cuts down on the verbosity of
parse_options_check().
In addition change the use of BUG() in that function to bug(), we'll
be dying soon enough, but always want exhaustive error reporting from
the function.
Let's also use bug() instead of BUG() in preprocess_options() and
parse_options_start_1() (which is called shortly after
preprocess_options()). Since the BUG_if_bug() is called at the end of
parse_options_start_1() we won't miss it, and even if we did the
invocation in common-main.c would trigger.
Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---
parse-options.c | 54 ++++++++++++++++++++++++-------------------------
1 file changed, 26 insertions(+), 28 deletions(-)
@@ -440,28 +440,27 @@ static void check_typos(const char *arg, const struct option *options)staticvoidparse_options_check(conststructoption*opts){-interr=0;charshort_opts[128];memset(short_opts,'\0',sizeof(short_opts));for(;opts->type!=OPTION_END;opts++){if((opts->flags&PARSE_OPT_LASTARG_DEFAULT)&&(opts->flags&PARSE_OPT_OPTARG))-err|=optbug(opts,"uses incompatible flags "-"LASTARG_DEFAULT and OPTARG");+optbug(opts,"uses incompatible flags "+"LASTARG_DEFAULT and OPTARG");if(opts->short_name){if(0x7F<=opts->short_name)-err|=optbug(opts,"invalid short name");+optbug(opts,"invalid short name");elseif(short_opts[opts->short_name]++)-err|=optbug(opts,"short name already used");+optbug(opts,"short name already used");}if(opts->flags&PARSE_OPT_NODASH&&((opts->flags&PARSE_OPT_OPTARG)||!(opts->flags&PARSE_OPT_NOARG)||!(opts->flags&PARSE_OPT_NONEG)||opts->long_name))-err|=optbug(opts,"uses feature "-"not supported for dashless options");+optbug(opts,"uses feature "+"not supported for dashless options");switch(opts->type){caseOPTION_COUNTUP:caseOPTION_BIT:
@@ -470,22 +469,22 @@ static void parse_options_check(const struct option *opts)caseOPTION_NUMBER:if((opts->flags&PARSE_OPT_OPTARG)||!(opts->flags&PARSE_OPT_NOARG))-err|=optbug(opts,"should not accept an argument");+optbug(opts,"should not accept an argument");break;caseOPTION_CALLBACK:if(!opts->callback&&!opts->ll_callback)-BUG("OPTION_CALLBACK needs one callback");+bug("OPTION_CALLBACK needs one callback");if(opts->callback&&opts->ll_callback)-BUG("OPTION_CALLBACK can't have two callbacks");+bug("OPTION_CALLBACK can't have two callbacks");break;caseOPTION_LOWLEVEL_CALLBACK:if(!opts->ll_callback)-BUG("OPTION_LOWLEVEL_CALLBACK needs a callback");+bug("OPTION_LOWLEVEL_CALLBACK needs a callback");if(opts->callback)-BUG("OPTION_LOWLEVEL_CALLBACK needs no high level callback");+bug("OPTION_LOWLEVEL_CALLBACK needs no high level callback");break;caseOPTION_ALIAS:-BUG("OPT_ALIAS() should not remain at this point. "+bug("OPT_ALIAS() should not remain at this point. ""Are you using parse_options_step() directly?\n""That case is not supported yet.");default:
@@ -493,10 +492,8 @@ static void parse_options_check(const struct option *opts)}if(opts->argh&&strcspn(opts->argh," _")!=strlen(opts->argh))-err|=optbug(opts,"multi-word argh should use dash to separate words");+optbug(opts,"multi-word argh should use dash to separate words");}-if(err)-exit(128);}staticvoidparse_options_start_1(structparse_opt_ctx_t*ctx,
@@ -518,11 +515,12 @@ static void parse_options_start_1(struct parse_opt_ctx_t *ctx,if((flags&PARSE_OPT_KEEP_UNKNOWN)&&(flags&PARSE_OPT_STOP_AT_NON_OPTION)&&!(flags&PARSE_OPT_ONE_SHOT))-BUG("STOP_AT_NON_OPTION and KEEP_UNKNOWN don't go together");+bug("STOP_AT_NON_OPTION and KEEP_UNKNOWN don't go together");if((flags&PARSE_OPT_ONE_SHOT)&&(flags&PARSE_OPT_KEEP_ARGV0))-BUG("Can't keep argv0 if you don't have it");+bug("Can't keep argv0 if you don't have it");parse_options_check(options);+BUG_if_bug();}voidparse_options_start(structparse_opt_ctx_t*ctx,
@@ -673,7 +671,7 @@ static struct option *preprocess_options(struct parse_opt_ctx_t *ctx,source=newopt[i].value;if(!long_name)-BUG("An alias must have long option name");+bug("An alias must have long option name");strbuf_addf(&help,_("alias of --%s"),source);for(j=0;j<nr;j++){
@@ -694,7 +692,7 @@ static struct option *preprocess_options(struct parse_opt_ctx_t *ctx,}if(j==nr)-BUG("could not find source option '%s' of alias '%s'",+bug("could not find source option '%s' of alias '%s'",source,newopt[i].long_name);ctx->alias_groups[alias*3+0]=newopt[i].long_name;ctx->alias_groups[alias*3+1]=options[j].long_name;
From: Taylor Blau <hidden> Date: 2021-11-16 18:44:05
On Mon, Nov 15, 2021 at 11:18:10PM +0100, Ævar Arnfjörð Bjarmason wrote:
Since everyone's getting in on the C99 fun.
Well, $subject and a bit more. This RFC series has bits and pieces
from thing I've submitted before. I'd proposed to make variadic macros
a hard dependency before in [1] because I wanted to get to the goal in
$subject, perhaps the whole thing will be more convincing.
This also includes the die_message() in a recent series of mine[2]
that I abandoned.
At the end of this series we expose a config variable to have
usage/die/warning emit line numbers. I.e. going from:
$ git -c core.usageAddSource=false -c core.x=y config --get --bool core.x
fatal: bad boolean config value 'y' for 'core.x'
To:
$ git -c core.usageAddSource=false -c core.x=y config --get --bool core.x
fatal: config.c:1241: bad boolean config value 'y' for 'core.x'
Just picking on this output change in particular. I agree that this is
easier for folks hacking on Git to trace down errors. But I'm not sure
that I could say then same about users, who will likely treat this extra
output as noise.
Now we may find it helpful if they include it in a bug report, but I
feel reasonably comfortable saying that the value there is pretty
marginal. I don't find it all that problematic to grep for a specific
error string, and usually find myself in the right place.
I find that to make tracing down errors in the test suite, and 21/21
has a GIT_TEST_* mode to turn it on there (which fails a lot now, but
I'm hoping I'll eventually get passing).
But most importantly we've now got meaningful file/line numbers in
trace2 error events. I.e. from all of them being some line in usage.c:
$ GIT_TRACE2_EVENT=/dev/stdout ~/g/git/git -c core.usageAddSource=false -c core.x=y config --get --bool core.x 2>&1 2>/dev/null|grep error | jq -r .
{
"event": "error",
"sid": "20211115T221343.534151Z-Hc2f5b994-P003f3980",
"thread": "main",
"time": "2021-11-15T22:13:43.537981Z",
"file": "usage.c",
"line": 65,
"msg": "bad boolean config value 'y' for 'core.x'",
"fmt": "bad boolean config value '%s' for '%s'"
}
To:
$ GIT_TRACE2_EVENT=/dev/stdout ~/g/git/git -c core.usageAddSource=false -c core.x=y config --get --bool core.x 2>&1 2>/dev/null|grep error | jq -r .
{
"event": "error",
"sid": "20211115T221357.083824Z-Hc2f5b994-P003f4a82",
"thread": "main",
"time": "2021-11-15T22:13:57.087596Z",
"file": "config.c",
"line": 1241,
"msg": "bad boolean config value 'y' for 'core.x'",
"fmt": "bad boolean config value '%s' for '%s'"
}
Neat. This is a use-case that has all of the value without putting it in
front of users all of the time. I like it.
This is "RFC" mainly because there's a CI failure in 0061.2 with this,
I still can't figure out what that's about (or if it's some fluke
unrelated to this topic), but that has to be investigated.
Hmm. Putting the CI failures aside for a second, wouldn't we want to
hold off on something like this until we have flown the C99 weather
balloon for a while? If we suddenly start introducing C99-isms into the
code while brian's patch is still young, then we can suddenly no longer
say, "oh, just drop this #if because there are no other C99-specific
uses here", and instead compilers that don't support the newer standard
are out of luck.
That may have been already communicated elsewhere in this message and/or
throughout your patch series, so if I missed it, I apologize. Just
felt that it was worth stating the obvious before we go too far down the
wrong path.
Thanks,
Taylor
On Mon, Nov 15, 2021 at 11:18:10PM +0100, Ævar Arnfjörð Bjarmason wrote:
quoted
Since everyone's getting in on the C99 fun.
Well, $subject and a bit more. This RFC series has bits and pieces
from thing I've submitted before. I'd proposed to make variadic macros
a hard dependency before in [1] because I wanted to get to the goal in
$subject, perhaps the whole thing will be more convincing.
This also includes the die_message() in a recent series of mine[2]
that I abandoned.
At the end of this series we expose a config variable to have
usage/die/warning emit line numbers. I.e. going from:
$ git -c core.usageAddSource=false -c core.x=y config --get --bool core.x
fatal: bad boolean config value 'y' for 'core.x'
To:
$ git -c core.usageAddSource=false -c core.x=y config --get --bool core.x
fatal: config.c:1241: bad boolean config value 'y' for 'core.x'
Just picking on this output change in particular. I agree that this is
easier for folks hacking on Git to trace down errors. But I'm not sure
that I could say then same about users, who will likely treat this extra
output as noise.
Now we may find it helpful if they include it in a bug report, but I
feel reasonably comfortable saying that the value there is pretty
marginal. I don't find it all that problematic to grep for a specific
error string, and usually find myself in the right place.
I wouldn't suggest exposing this to users, except perhaps as part of
some "how to submit a bugreport" instructions. It's thoroughly optional.
I thought it was easy enough to do with the preceding steps since all
the data is there, and would help my workflow a lot.
If you've got the file/line number like that you can make it clickable
in your terminal/compile mode, e.g. Emacs's M-x compile. Saves time over
having to grep manually select the string, grep for it etc.
Anyway, I can certainly live with peeling this patch off the end and
just stopping at the trace2 data for now, if you/others feel strongly
about it.
quoted
I find that to make tracing down errors in the test suite, and 21/21
has a GIT_TEST_* mode to turn it on there (which fails a lot now, but
I'm hoping I'll eventually get passing).
But most importantly we've now got meaningful file/line numbers in
trace2 error events. I.e. from all of them being some line in usage.c:
$ GIT_TRACE2_EVENT=/dev/stdout ~/g/git/git -c core.usageAddSource=false -c core.x=y config --get --bool core.x 2>&1 2>/dev/null|grep error | jq -r .
{
"event": "error",
"sid": "20211115T221343.534151Z-Hc2f5b994-P003f3980",
"thread": "main",
"time": "2021-11-15T22:13:43.537981Z",
"file": "usage.c",
"line": 65,
"msg": "bad boolean config value 'y' for 'core.x'",
"fmt": "bad boolean config value '%s' for '%s'"
}
To:
$ GIT_TRACE2_EVENT=/dev/stdout ~/g/git/git -c core.usageAddSource=false -c core.x=y config --get --bool core.x 2>&1 2>/dev/null|grep error | jq -r .
{
"event": "error",
"sid": "20211115T221357.083824Z-Hc2f5b994-P003f4a82",
"thread": "main",
"time": "2021-11-15T22:13:57.087596Z",
"file": "config.c",
"line": 1241,
"msg": "bad boolean config value 'y' for 'core.x'",
"fmt": "bad boolean config value '%s' for '%s'"
}
Neat. This is a use-case that has all of the value without putting it in
front of users all of the time. I like it.
quoted
This is "RFC" mainly because there's a CI failure in 0061.2 with this,
I still can't figure out what that's about (or if it's some fluke
unrelated to this topic), but that has to be investigated.
Hmm. Putting the CI failures aside for a second, wouldn't we want to
hold off on something like this until we have flown the C99 weather
balloon for a while? If we suddenly start introducing C99-isms into the
code while brian's patch is still young, then we can suddenly no longer
say, "oh, just drop this #if because there are no other C99-specific
uses here", and instead compilers that don't support the newer standard
are out of luck.
That may have been already communicated elsewhere in this message and/or
throughout your patch series, so if I missed it, I apologize. Just
felt that it was worth stating the obvious before we go too far down the
wrong path.
As noted in 02/21 we're hard depending on this particular C99 feature
already fon a few releases now, the only change on that front in this
series is to stop committing to maintaining the non-C99 codepaths.
We've already had hard dependencies on various bits of C99 for years now
without any trouble, and I wouldn't expect any problems on this front
either.
Brian's series and the current weatherbaloon from Junio are a bit
different in trying out new things we either haven't done before, or
have run into some trouble with in the past.
No need at all to apologize, it's a lot of patches, and raising this
sort of thing is what patch review is good for.
Thanks a lot for looking this over.
From: Taylor Blau <hidden> Date: 2021-11-16 19:36:51
On Tue, Nov 16, 2021 at 07:58:01PM +0100, Ævar Arnfjörð Bjarmason wrote:
On Tue, Nov 16 2021, Taylor Blau wrote:
quoted
quoted
At the end of this series we expose a config variable to have
usage/die/warning emit line numbers. I.e. going from:
$ git -c core.usageAddSource=false -c core.x=y config --get --bool core.x
fatal: bad boolean config value 'y' for 'core.x'
To:
$ git -c core.usageAddSource=false -c core.x=y config --get --bool core.x
fatal: config.c:1241: bad boolean config value 'y' for 'core.x'
Just picking on this output change in particular. I agree that this is
easier for folks hacking on Git to trace down errors. But I'm not sure
that I could say then same about users, who will likely treat this extra
output as noise.
Now we may find it helpful if they include it in a bug report, but I
feel reasonably comfortable saying that the value there is pretty
marginal. I don't find it all that problematic to grep for a specific
error string, and usually find myself in the right place.
I wouldn't suggest exposing this to users, except perhaps as part of
some "how to submit a bugreport" instructions. It's thoroughly optional.
I thought it was easy enough to do with the preceding steps since all
the data is there, and would help my workflow a lot.
If you've got the file/line number like that you can make it clickable
in your terminal/compile mode, e.g. Emacs's M-x compile. Saves time over
having to grep manually select the string, grep for it etc.
Anyway, I can certainly live with peeling this patch off the end and
just stopping at the trace2 data for now, if you/others feel strongly
about it.
I don't feel strongly, and I was just noting that it seemed like users
would treat this extra information more often as noise than anything
else.
When you talk about making it optional, do you mean through
configuration / an environment variable, or by including / not including
the patch? In other words, the latter seems much more like us making a
decision on whether or not to include line numbers rather than
presenting a new option to users, though I may be misunderstanding.
As noted in 02/21 we're hard depending on this particular C99 feature
already fon a few releases now, the only change on that front in this
series is to stop committing to maintaining the non-C99 codepaths.
We've already had hard dependencies on various bits of C99 for years now
without any trouble, and I wouldn't expect any problems on this front
either.
Interesting; so this and others are likely part of MSVC's kind-of
support for C99 features? In other words, that MSVC supports some
features from C99 (and we are depending on a subset of those) but not
all features so that it could reasonably be called a spec-compliant
compiler for the C99 standard? If so, makes sense.
Thanks,
Taylor
On Tue, Nov 16, 2021 at 07:58:01PM +0100, Ævar Arnfjörð Bjarmason wrote:
quoted
On Tue, Nov 16 2021, Taylor Blau wrote:
quoted
quoted
At the end of this series we expose a config variable to have
usage/die/warning emit line numbers. I.e. going from:
$ git -c core.usageAddSource=false -c core.x=y config --get --bool core.x
fatal: bad boolean config value 'y' for 'core.x'
To:
$ git -c core.usageAddSource=false -c core.x=y config --get --bool core.x
fatal: config.c:1241: bad boolean config value 'y' for 'core.x'
Just picking on this output change in particular. I agree that this is
easier for folks hacking on Git to trace down errors. But I'm not sure
that I could say then same about users, who will likely treat this extra
output as noise.
Now we may find it helpful if they include it in a bug report, but I
feel reasonably comfortable saying that the value there is pretty
marginal. I don't find it all that problematic to grep for a specific
error string, and usually find myself in the right place.
I wouldn't suggest exposing this to users, except perhaps as part of
some "how to submit a bugreport" instructions. It's thoroughly optional.
I thought it was easy enough to do with the preceding steps since all
the data is there, and would help my workflow a lot.
If you've got the file/line number like that you can make it clickable
in your terminal/compile mode, e.g. Emacs's M-x compile. Saves time over
having to grep manually select the string, grep for it etc.
Anyway, I can certainly live with peeling this patch off the end and
just stopping at the trace2 data for now, if you/others feel strongly
about it.
I don't feel strongly, and I was just noting that it seemed like users
would treat this extra information more often as noise than anything
else.
When you talk about making it optional, do you mean through
configuration / an environment variable, or by including / not including
the patch? In other words, the latter seems much more like us making a
decision on whether or not to include line numbers rather than
presenting a new option to users, though I may be misunderstanding.
Not surprising, since I see that I screwed up the summary in both the CL
and 21/21. I.e. both of these are =false (I copy/pasted the error
around, but didn't adjust the command that was invoked):
$ git -c core.usageAddSource=false -c core.x=y config --get --bool core.x
fatal: bad boolean config value 'y' for 'core.x'
$ git -c core.usageAddSource=false -c core.x=y config --get --bool core.x
fatal: config.c:1241: bad boolean config value 'y' for 'core.x'
That second one should be core.usageAddSource=true, i.e. as seen in the
21/21 implentation and core.txt docs you only get these line numebers if
you opt-in to them via configuration or the new GIT_TEST_* environment
variable.
quoted
As noted in 02/21 we're hard depending on this particular C99 feature
already fon a few releases now, the only change on that front in this
series is to stop committing to maintaining the non-C99 codepaths.
quoted
We've already had hard dependencies on various bits of C99 for years now
without any trouble, and I wouldn't expect any problems on this front
either.
Interesting; so this and others are likely part of MSVC's kind-of
support for C99 features? In other words, that MSVC supports some
features from C99 (and we are depending on a subset of those) but not
all features so that it could reasonably be called a spec-compliant
compiler for the C99 standard? If so, makes sense.
Yes, I think this is one of the things that's the same or similar enough
to C++ that MSVC has good support for it. See the "We try to support a
wide range of C compilers[...]" section in the CodingGuidelines for a
list of some other C99 features we've used for years already.
From: Jeff Hostetler <hidden> Date: 2021-12-27 19:32:45
On 11/15/21 5:18 PM, Ævar Arnfjörð Bjarmason wrote:
[...]
It might be a good change to remove the "fmt" key from the "error"
events as a follow-up change. As these few examples from running the
test suite show it's sometimes redundant (same as the "msg"), rather
useless (just a "%s"), or something we could now mostly aggregate by
file/line instead of the normalized printf format:
1 file":"builtin/gc.c","line":1391,"msg":"'bogus' is not a valid task","fmt":"'%s' is not a valid task"}
1 file":"builtin/for-each-ref.c","line":89,"msg":"format: %(then) atom used more than once","fmt":"%s"}
1 file":"builtin/fast-import.c","line":411,"msg":"Garbage after mark: N :202 :302x","fmt":"Garbage after mark: %s"}
"Mostly" here assumes that it would be OK if the aggregation changed
between git versions, which may be what all users of trace2 want. The
change that introduced the "fmt" code was ee4512ed481 (trace2: create
new combined trace facility, 2019-02-22), and the documentation change
was e544221d97a (trace2: Documentation/technical/api-trace2.txt,
2019-02-22).
Both are rather vague on what problem "fmt" solved exactly, aside from
the obvious one of it being impossible to do meaningful aggregations
due to the "file" and "line" being the same everywhere, which isn't
the case now.
In any case, let's leave "fmt" be for now, the above summary was given
in case it's interesting to remove it in the future, e.g. to save
space in trace2 payloads.
I added the "fmt" field so that we could do aggregations
of error messages across multiple users without regard
to what branch or filename or percentage or whatever was
formatted into the actual "msg" written to stderr.
The actual file:line wasn't useful (primarily because it
was probably something in usage.c), but even if we fix that
it might not be useful if we have users running 10 different
versions of Git (because some people don't upgrade immediately).
So I'd rather not kill it right now.
Jeff
On 11/15/21 5:18 PM, Ævar Arnfjörð Bjarmason wrote:
[...]
quoted
It might be a good change to remove the "fmt" key from the "error"
events as a follow-up change. As these few examples from running the
test suite show it's sometimes redundant (same as the "msg"), rather
useless (just a "%s"), or something we could now mostly aggregate by
file/line instead of the normalized printf format:
1 file":"builtin/gc.c","line":1391,"msg":"'bogus' is not a
valid task","fmt":"'%s' is not a valid task"}
1 file":"builtin/for-each-ref.c","line":89,"msg":"format: %(then) atom used more than once","fmt":"%s"}
1 file":"builtin/fast-import.c","line":411,"msg":"Garbage after mark: N :202 :302x","fmt":"Garbage after mark: %s"}
"Mostly" here assumes that it would be OK if the aggregation changed
between git versions, which may be what all users of trace2 want. The
change that introduced the "fmt" code was ee4512ed481 (trace2: create
new combined trace facility, 2019-02-22), and the documentation change
was e544221d97a (trace2: Documentation/technical/api-trace2.txt,
2019-02-22).
Both are rather vague on what problem "fmt" solved exactly, aside
from
the obvious one of it being impossible to do meaningful aggregations
due to the "file" and "line" being the same everywhere, which isn't
the case now.
In any case, let's leave "fmt" be for now, the above summary was
given
in case it's interesting to remove it in the future, e.g. to save
space in trace2 payloads.
I added the "fmt" field so that we could do aggregations
of error messages across multiple users without regard
to what branch or filename or percentage or whatever was
formatted into the actual "msg" written to stderr.
The actual file:line wasn't useful (primarily because it
was probably something in usage.c), but even if we fix that
it might not be useful if we have users running 10 different
versions of Git (because some people don't upgrade immediately).
So I'd rather not kill it right now.
Thanks. I'm not trying to kill it, but just poking at what it was for
exactly.
Depending on the answer to that perhaps we didn't need it anymore, but
the explanation you provide (mostly) makes sense.
The "mostly" being because I'm assuming that you only need to deal with
LC_ALL=C users?
I.e. the documented promise that you can group things by "fmt" doesn't
hold if you're processing even streams from users who are using a
translated git, because we'll get the translated format string, not the
original.
For that we'd need to change the API from/to to:
- error(_("some format %s"), ...)
+ error(N_("some format %s"), ...)
So being able to say "just group on file/line" would be simpler.
And also "mostly" because the "fmt" case also won't handle these and
other duplicate formats (but maybe you haven't run into them in
practice):
$ git grep -E '\b(usage|die|error|warning)(_errno)?\("%s\"' -- '*.[ch]' | wc -l
90
So I was somewhat hoping for future work that you'd be OK with the new
file/line grouping.
Because keeping "fmt" would eventually need some massive coccinelle
search/replacement for "_(...)" -> "N_(...)" per the above, even then
consumers of the stream would get duplicate grouping for the likes of
"%s".
Do you think if as a follow-up we had "__func__"[1] along with
"file/line" that the "file/__func__" combination would be good enough?
The advantage of that would be that we could punt that "fmt"
change/complexity and document:
If you'd like to group errors the "file/line" pair will be unique
enough within a given git version to do so (sans a few codepaths that
relay errors from elsewhere).
If you'd like a semi-stable grouping across similar git versions the
"file/func" pair should be Good Enough for most purposes. Some functions
might emit multiple errors, but you'd probably want to group them as
similar enough anyway.
But I realize that those things don't give you exactly the same things
that "fmt" does, but maybe they're good enough (or even better?), or
not.
1. https://gcc.gnu.org/onlinedocs/gcc/Function-Names.html
From: Jeff Hostetler <hidden> Date: 2021-12-28 16:32:45
On 12/27/21 6:01 PM, Ævar Arnfjörð Bjarmason wrote:
On Mon, Dec 27 2021, Jeff Hostetler wrote:
quoted
On 11/15/21 5:18 PM, Ævar Arnfjörð Bjarmason wrote:
[...]
quoted
It might be a good change to remove the "fmt" key from the "error"
events as a follow-up change. As these few examples from running the
test suite show it's sometimes redundant (same as the "msg"), rather
useless (just a "%s"), or something we could now mostly aggregate by
file/line instead of the normalized printf format:
1 file":"builtin/gc.c","line":1391,"msg":"'bogus' is not a
valid task","fmt":"'%s' is not a valid task"}
1 file":"builtin/for-each-ref.c","line":89,"msg":"format: %(then) atom used more than once","fmt":"%s"}
1 file":"builtin/fast-import.c","line":411,"msg":"Garbage after mark: N :202 :302x","fmt":"Garbage after mark: %s"}
"Mostly" here assumes that it would be OK if the aggregation changed
between git versions, which may be what all users of trace2 want. The
change that introduced the "fmt" code was ee4512ed481 (trace2: create
new combined trace facility, 2019-02-22), and the documentation change
was e544221d97a (trace2: Documentation/technical/api-trace2.txt,
2019-02-22).
Both are rather vague on what problem "fmt" solved exactly, aside
from
the obvious one of it being impossible to do meaningful aggregations
due to the "file" and "line" being the same everywhere, which isn't
the case now.
In any case, let's leave "fmt" be for now, the above summary was
given
in case it's interesting to remove it in the future, e.g. to save
space in trace2 payloads.
I added the "fmt" field so that we could do aggregations
of error messages across multiple users without regard
to what branch or filename or percentage or whatever was
formatted into the actual "msg" written to stderr.
The actual file:line wasn't useful (primarily because it
was probably something in usage.c), but even if we fix that
it might not be useful if we have users running 10 different
versions of Git (because some people don't upgrade immediately).
So I'd rather not kill it right now.
Thanks. I'm not trying to kill it, but just poking at what it was for
exactly.
Depending on the answer to that perhaps we didn't need it anymore, but
the explanation you provide (mostly) makes sense.
The "mostly" being because I'm assuming that you only need to deal with
LC_ALL=C users?
I.e. the documented promise that you can group things by "fmt" doesn't
hold if you're processing even streams from users who are using a
translated git, because we'll get the translated format string, not the
original.
I just did a query on the data we've collected over the last
few weeks and there are only English error messages in the
database, so yes LC_ALL=C seems to be the norm.
For that we'd need to change the API from/to to:
- error(_("some format %s"), ...)
+ error(N_("some format %s"), ...)
So no, I don't think it is worth the complexity to change
this. Besides, wouldn't you need to more machinery under
the hood -- to emit the untranslated string to trace2 and
the translated string to stderr? (As in, move the translation
down a layer??)
My "fmt" field is not worth that effort.
And besides, my goal was only to get the "top 10 or 20 errors"
across a large set of users. I guess I'm thinking of it as a
sample rather than an exhaustive list, so it is OK if we don't
capture the translated strings.
Something else to consider is the GDPR. The "fmt" string is
generic (e.g. "path '%s' exists on disk, but not in the index")
but doesn't leak an PII or otherwise sensitive data. Whereas
the corresponding "msg" field does include the pathname in this
example. So if someone is post-processing the data and aggregating,
they may want to relay only the "fmt" field and not the "msg" field
to their database.
(Granted, there are lots of PII and GDPR problematic fields in
the data stream that a post-processor would need to be aware of,
but all of that is outside of the scope of the Trace2 logging.
I only mention it here because the "fmt" field may be useful for
reasons not previously discussed.)
So being able to say "just group on file/line" would be simpler.
And also "mostly" because the "fmt" case also won't handle these and
other duplicate formats (but maybe you haven't run into them in
practice):
$ git grep -E '\b(usage|die|error|warning)(_errno)?\("%s\"' -- '*.[ch]' | wc -l
90
So I was somewhat hoping for future work that you'd be OK with the new
file/line grouping.
Because keeping "fmt" would eventually need some massive coccinelle
search/replacement for "_(...)" -> "N_(...)" per the above, even then
consumers of the stream would get duplicate grouping for the likes of
"%s".
Do you think if as a follow-up we had "__func__"[1] along with
"file/line" that the "file/__func__" combination would be good enough?
The advantage of that would be that we could punt that "fmt"
change/complexity and document:
If you'd like to group errors the "file/line" pair will be unique
enough within a given git version to do so (sans a few codepaths that
relay errors from elsewhere).
If you'd like a semi-stable grouping across similar git versions the
"file/func" pair should be Good Enough for most purposes. Some functions
might emit multiple errors, but you'd probably want to group them as
similar enough anyway.
But I realize that those things don't give you exactly the same things
that "fmt" does, but maybe they're good enough (or even better?), or
not.
1. https://gcc.gnu.org/onlinedocs/gcc/Function-Names.html
I'll have to think about this some and get back to you.
Jeff
Just adding another datapoint...
At my $DAYJOB, we've been collecting trace2 information but have only
used it very lightly so far. We've been meaning to use it more.
Anyway, I have a couple (possibly not fully informed) opinions based
on that...
On Tue, Dec 28, 2021 at 8:32 AM Jeff Hostetler [off-list ref] wrote:
On 12/27/21 6:01 PM, Ævar Arnfjörð Bjarmason wrote:
quoted
On Mon, Dec 27 2021, Jeff Hostetler wrote:
quoted
On 11/15/21 5:18 PM, Ævar Arnfjörð Bjarmason wrote:
[...]
quoted
It might be a good change to remove the "fmt" key from the "error"
events as a follow-up change. As these few examples from running the
test suite show it's sometimes redundant (same as the "msg"), rather
useless (just a "%s"), or something we could now mostly aggregate by
file/line instead of the normalized printf format:
1 file":"builtin/gc.c","line":1391,"msg":"'bogus' is not a
valid task","fmt":"'%s' is not a valid task"}
1 file":"builtin/for-each-ref.c","line":89,"msg":"format: %(then) atom used more than once","fmt":"%s"}
1 file":"builtin/fast-import.c","line":411,"msg":"Garbage after mark: N :202 :302x","fmt":"Garbage after mark: %s"}
"Mostly" here assumes that it would be OK if the aggregation changed
between git versions, which may be what all users of trace2 want. The
change that introduced the "fmt" code was ee4512ed481 (trace2: create
new combined trace facility, 2019-02-22), and the documentation change
was e544221d97a (trace2: Documentation/technical/api-trace2.txt,
2019-02-22).
Both are rather vague on what problem "fmt" solved exactly, aside
from
the obvious one of it being impossible to do meaningful aggregations
due to the "file" and "line" being the same everywhere, which isn't
the case now.
In any case, let's leave "fmt" be for now, the above summary was
given
in case it's interesting to remove it in the future, e.g. to save
space in trace2 payloads.
I added the "fmt" field so that we could do aggregations
of error messages across multiple users without regard
to what branch or filename or percentage or whatever was
formatted into the actual "msg" written to stderr.
The actual file:line wasn't useful (primarily because it
was probably something in usage.c), but even if we fix that
it might not be useful if we have users running 10 different
versions of Git (because some people don't upgrade immediately).
So I'd rather not kill it right now.
Thanks. I'm not trying to kill it, but just poking at what it was for
exactly.
Depending on the answer to that perhaps we didn't need it anymore, but
the explanation you provide (mostly) makes sense.
The "mostly" being because I'm assuming that you only need to deal with
LC_ALL=C users?
I.e. the documented promise that you can group things by "fmt" doesn't
hold if you're processing even streams from users who are using a
translated git, because we'll get the translated format string, not the
original.
I just did a query on the data we've collected over the last
few weeks and there are only English error messages in the
database, so yes LC_ALL=C seems to be the norm.
quoted
For that we'd need to change the API from/to to:
- error(_("some format %s"), ...)
+ error(N_("some format %s"), ...)
So no, I don't think it is worth the complexity to change
this. Besides, wouldn't you need to more machinery under
the hood -- to emit the untranslated string to trace2 and
the translated string to stderr? (As in, move the translation
down a layer??)
My "fmt" field is not worth that effort.
And besides, my goal was only to get the "top 10 or 20 errors"
across a large set of users. I guess I'm thinking of it as a
sample rather than an exhaustive list, so it is OK if we don't
capture the translated strings.
This makes sense to me.
Something else to consider is the GDPR. The "fmt" string is
generic (e.g. "path '%s' exists on disk, but not in the index")
but doesn't leak an PII or otherwise sensitive data. Whereas
the corresponding "msg" field does include the pathname in this
example. So if someone is post-processing the data and aggregating,
they may want to relay only the "fmt" field and not the "msg" field
to their database.
(Granted, there are lots of PII and GDPR problematic fields in
the data stream that a post-processor would need to be aware of,
but all of that is outside of the scope of the Trace2 logging.
I only mention it here because the "fmt" field may be useful for
reasons not previously discussed.)
quoted
So being able to say "just group on file/line" would be simpler.
And also "mostly" because the "fmt" case also won't handle these and
other duplicate formats (but maybe you haven't run into them in
practice):
$ git grep -E '\b(usage|die|error|warning)(_errno)?\("%s\"' -- '*.[ch]' | wc -l
90
So I was somewhat hoping for future work that you'd be OK with the new
file/line grouping.
Because keeping "fmt" would eventually need some massive coccinelle
search/replacement for "_(...)" -> "N_(...)" per the above, even then
consumers of the stream would get duplicate grouping for the likes of
"%s".
Do you think if as a follow-up we had "__func__"[1] along with
"file/line" that the "file/__func__" combination would be good enough?
The advantage of that would be that we could punt that "fmt"
change/complexity and document:
If you'd like to group errors the "file/line" pair will be unique
enough within a given git version to do so (sans a few codepaths that
relay errors from elsewhere).
I don't actually like the file/line pair technique so much. We have a
little influence but no control over deployed git versions. We've
collected data about git versions in use, and we've got every single
git version going back several years in use (plus, some people took
the copy of Git I built with ort and other stuff included early but
they haven't all upgraded as I've updated it leaving us with an even
greater sprawl of file/line numbers). That means file/line pairs
would split messages much more than the number of languages would.
(We do have developers in over a dozen different countries, though we
have a strong majority in English-speaking countries.)
quoted
If you'd like a semi-stable grouping across similar git versions the
"file/func" pair should be Good Enough for most purposes. Some functions
might emit multiple errors, but you'd probably want to group them as
similar enough anyway.
Why would we want to group different errors? Isn't the point to
figure out which error is being triggered the most (or which errors)?
This sounds like it'd leave us with more investigation work to do.
quoted
But I realize that those things don't give you exactly the same things
that "fmt" does, but maybe they're good enough (or even better?), or
not.
On 12/27/21 6:01 PM, Ævar Arnfjörð Bjarmason wrote:
quoted
On Mon, Dec 27 2021, Jeff Hostetler wrote:
quoted
On 11/15/21 5:18 PM, Ævar Arnfjörð Bjarmason wrote:
[...]
quoted
It might be a good change to remove the "fmt" key from the "error"
events as a follow-up change. As these few examples from running the
test suite show it's sometimes redundant (same as the "msg"), rather
useless (just a "%s"), or something we could now mostly aggregate by
file/line instead of the normalized printf format:
1 file":"builtin/gc.c","line":1391,"msg":"'bogus' is not a
valid task","fmt":"'%s' is not a valid task"}
1 file":"builtin/for-each-ref.c","line":89,"msg":"format: %(then) atom used more than once","fmt":"%s"}
1 file":"builtin/fast-import.c","line":411,"msg":"Garbage after mark: N :202 :302x","fmt":"Garbage after mark: %s"}
"Mostly" here assumes that it would be OK if the aggregation changed
between git versions, which may be what all users of trace2 want. The
change that introduced the "fmt" code was ee4512ed481 (trace2: create
new combined trace facility, 2019-02-22), and the documentation change
was e544221d97a (trace2: Documentation/technical/api-trace2.txt,
2019-02-22).
Both are rather vague on what problem "fmt" solved exactly, aside
from
the obvious one of it being impossible to do meaningful aggregations
due to the "file" and "line" being the same everywhere, which isn't
the case now.
In any case, let's leave "fmt" be for now, the above summary was
given
in case it's interesting to remove it in the future, e.g. to save
space in trace2 payloads.
I added the "fmt" field so that we could do aggregations
of error messages across multiple users without regard
to what branch or filename or percentage or whatever was
formatted into the actual "msg" written to stderr.
The actual file:line wasn't useful (primarily because it
was probably something in usage.c), but even if we fix that
it might not be useful if we have users running 10 different
versions of Git (because some people don't upgrade immediately).
So I'd rather not kill it right now.
Thanks. I'm not trying to kill it, but just poking at what it was
for
exactly.
Depending on the answer to that perhaps we didn't need it anymore,
but
the explanation you provide (mostly) makes sense.
The "mostly" being because I'm assuming that you only need to deal
with
LC_ALL=C users?
I.e. the documented promise that you can group things by "fmt"
doesn't
hold if you're processing even streams from users who are using a
translated git, because we'll get the translated format string, not the
original.
I just did a query on the data we've collected over the last
few weeks and there are only English error messages in the
database, so yes LC_ALL=C seems to be the norm.
Ah, that explains it.
quoted
For that we'd need to change the API from/to to:
- error(_("some format %s"), ...)
+ error(N_("some format %s"), ...)
So no, I don't think it is worth the complexity to change
this. Besides, wouldn't you need to more machinery under
the hood -- to emit the untranslated string to trace2 and
the translated string to stderr? (As in, move the translation
down a layer??)
Yes, hence N_(), it marks the string for translation, but doesn't
translate it. So the underlying function would call _() for what we emit
to stderr, but not for the "fmt" field.
My "fmt" field is not worth that effort.
Probably not to you because you're deploying git into a monolingual
environment, but for anyone who is they'd need to maintain manual
groupings of messages by scraping our po/*.po files.
And besides, my goal was only to get the "top 10 or 20 errors"
across a large set of users. I guess I'm thinking of it as a
sample rather than an exhaustive list, so it is OK if we don't
capture the translated strings.
I suppose with enough users it wouldn't matter either way, but you'd get
quite a bit of fragmentation. You'd also have errors that don't differ
between translations (e.g. those "%s" cases) amplified in count, as they
won't fragment due to i18n.
Something else to consider is the GDPR. The "fmt" string is
generic (e.g. "path '%s' exists on disk, but not in the index")
but doesn't leak an PII or otherwise sensitive data. Whereas
the corresponding "msg" field does include the pathname in this
example. So if someone is post-processing the data and aggregating,
they may want to relay only the "fmt" field and not the "msg" field
to their database.
Indeed.
(Granted, there are lots of PII and GDPR problematic fields in
the data stream that a post-processor would need to be aware of,
but all of that is outside of the scope of the Trace2 logging.
I only mention it here because the "fmt" field may be useful for
reasons not previously discussed.)
I do think it would be a good thing to work on, i.e. to make the logging
less verbose, which as a side-effect would make it GDPR compliant.
On Tue, Dec 28, 2021 at 8:32 AM Jeff Hostetler [off-list ref] wrote:
quoted
quoted
If you'd like a semi-stable grouping across similar git versions the
"file/func" pair should be Good Enough for most purposes. Some functions
might emit multiple errors, but you'd probably want to group them as
similar enough anyway.
Why would we want to group different errors? Isn't the point to
figure out which error is being triggered the most (or which errors)?
This sounds like it'd leave us with more investigation work to do.
Ideally you wouldn't, i.e. the goal here is to get some approximation of
a unique ID for an error across versions.
But unless we're going to assign something like MySQL's error ID's
manually any automatic method we pick is only going to be an
approximation.
So the question is whether we can have something that's good enough. The
current "fmt" feature is fragmented by i18n. That's fixable (at the cost
of quite a lot of lines changed), but would something even more succinct
be good enough?
Which is why I suggested file/function, i.e. it'll have some
duplication, but for an error dashboard using trace2 data I'd think it's
probably good enough.
But maybe not. I just wanted to ask about it as a quick question...
quoted
quoted
But I realize that those things don't give you exactly the same things
that "fmt" does, but maybe they're good enough (or even better?), or
not.
On Tue, Dec 28, 2021 at 3:53 PM Ævar Arnfjörð Bjarmason
[off-list ref] wrote:
On Tue, Dec 28 2021, Elijah Newren wrote:
quoted
On Tue, Dec 28, 2021 at 8:32 AM Jeff Hostetler [off-list ref] wrote:
quoted
quoted
If you'd like a semi-stable grouping across similar git versions the
"file/func" pair should be Good Enough for most purposes. Some functions
might emit multiple errors, but you'd probably want to group them as
similar enough anyway.
Why would we want to group different errors? Isn't the point to
figure out which error is being triggered the most (or which errors)?
This sounds like it'd leave us with more investigation work to do.
Ideally you wouldn't, i.e. the goal here is to get some approximation of
a unique ID for an error across versions.
But unless we're going to assign something like MySQL's error ID's
manually any automatic method we pick is only going to be an
approximation.
I like this way that you frame it. I agree.
So the question is whether we can have something that's good enough. The
current "fmt" feature is fragmented by i18n. That's fixable (at the cost
of quite a lot of lines changed), but would something even more succinct
be good enough?
Which is why I suggested file/function, i.e. it'll have some
duplication, but for an error dashboard using trace2 data I'd think it's
probably good enough.
But maybe not. I just wanted to ask about it as a quick question...
I think for determining the most frequently triggered errors,
fragmentation is a minor issue, so you are right to call it out. In
particular, having the counts of issues separated by language might
mean that when we pick the top N errors, some of those in the top N
wouldn't really be in the top N if we had them correctly combined with
the other translations (and we also might get duplicates within our
chosen top N, since an english and a german translation of the same
error are both in the top N of the fragmented counts). Pretty
unlikely to be a problem in practice, though, and rather trivial to
work around once we have the data collected and are looking into it.
Even in the really unlikely event that I was trying to fix a "top N"
problem and accidentally ended up with a "top N+2" problem, I'm still
dealing with a "real error" that users are hitting. Any work I do to
fix it will help people facing a real problem.
In contrast, coalescing of errors to me would be a major issue. Let's
say I look at the top error, as reported by file/function. But that
one error is from a function that has four error paths. If I take a
guess at one of those error paths and try to fix it, I might be
chasing ghosts and completely wasting my time. My first step should
be to go back to the drawing board and attempt to collect data about
what error the user was actually hitting (a rather lengthy process,
especially in attempting over a period of weeks/months to cajole users
to upgrade their git versions to get the new logging) -- but that was
exactly what this trace2 stuff was supposed to be doing in the first
place, so the file/function approximation choice defeats the purpose
of this error logging. It sounds like a deal breaker to me.
My gut instinct is that I'd take nearly any level of fragmentation
over the possible coalescing of separate errors.
I think the fragmentation solutions probably fall under the "good
enough" category. So, for example, the file/line number might be good
enough. It's a lot more fragmentation than different languages,
though, and it also suffers from the problem that it's hard to tell if
new git versions are fixing some of the "top N" problems (because new
git versions would have different line numbers and thus represent the
top N problems differently, whereas the fmt-based fragmentation will
at least be relatively consistent in its representation of errors
across git versions). But if the fmt solution was super problematic
for some other reasons, I'd gladly take file/line-number over
file/function.
So, of the solutions presented so far, the "fmt" feature seems to me
to be the best reasonable effort approximation.
Anyway, just my $0.02...
From: Jeff Hostetler <hidden> Date: 2021-12-29 16:13:08
On 12/28/21 11:32 AM, Jeff Hostetler wrote:
On 12/27/21 6:01 PM, Ævar Arnfjörð Bjarmason wrote:
quoted
On Mon, Dec 27 2021, Jeff Hostetler wrote:
quoted
On 11/15/21 5:18 PM, Ævar Arnfjörð Bjarmason wrote:
[...]
So being able to say "just group on file/line" would be simpler.
And also "mostly" because the "fmt" case also won't handle these and
other duplicate formats (but maybe you haven't run into them in
practice):
$ git grep -E '\b(usage|die|error|warning)(_errno)?\("%s\"' --
'*.[ch]' | wc -l
90
So I was somewhat hoping for future work that you'd be OK with the new
file/line grouping.
Because keeping "fmt" would eventually need some massive coccinelle
search/replacement for "_(...)" -> "N_(...)" per the above, even then
consumers of the stream would get duplicate grouping for the likes of
"%s".
Do you think if as a follow-up we had "__func__"[1] along with
"file/line" that the "file/__func__" combination would be good enough?
The advantage of that would be that we could punt that "fmt"
change/complexity and document:
If you'd like to group errors the "file/line" pair will be unique
enough within a given git version to do so (sans a few codepaths
that
relay errors from elsewhere).
If you'd like a semi-stable grouping across similar git versions the
"file/func" pair should be Good Enough for most purposes. Some
functions
might emit multiple errors, but you'd probably want to group them as
similar enough anyway.
But I realize that those things don't give you exactly the same things
that "fmt" does, but maybe they're good enough (or even better?), or
not.
1. https://gcc.gnu.org/onlinedocs/gcc/Function-Names.html
I'll have to think about this some and get back to you.
I think I'd rather just have the "fmt" string in the log.
I don't think the file:line or func:line helps here. Elsewhere
in this thread we've talked about having to support a user base
running various versions of Git (and don't forget to mix in GFW
and the GVFS-enabled version of Git).
Keep in mind that some users enable (or their EngSys/IT team
enables for them) brief mode (GIT_TRACE2_EVENT_BRIEF)
which omits the file:line data from the log. This reduces the
size of the data stream on all events, so we don't have that
data for some users -- and forcing it on would send a lot more
data and cost a lot more than any savings from omitting the
somewhat redundant "fmt" field.
Just having the format string usually lets us track down the
error/die call in the code using just grep (unless it is one of
those `die("%s",...)` cases (which should be fixed independent
of this discussion)). As Elijah mentioned elsewhere in this
thread, just having the format string doesn't mean we know exactly
which path/branch in the code lead us to that error, so local
analysis is still required if we want to kill a "top 10" item.
Having file:line doesn't help with that local effort.
As for handling translations, I'm not really worried about it.
If a post-processor really wants to get complete aggregations
independent of the users locale they could maybe build a tool
to load up the .po files, build a reverse index, and add a table
to their database that they could join with. I'm speculating
here that anyone would want to go to that trouble, but it is
possible. And it would keep all of this churn out of Git code.
Coalescing the format strings is the simplest by far. So I'd
really like to leave things the way they are.
Jeff