From: Jeff Hostetler via GitGitGadget <hidden> Date: 2021-09-15 20:36:26
In July I sent the V3 version [3] of my Builtin FSMonitor series [1,2,3].
This has been in "seen" as branch jh/builtin-fsmonitor. At 34 commits, it
was already a little too big to easily review. With the feed back on [3] and
from people using the experimental version that we shipped with
git-for-windows v2.32 and v2.33, I've prepared a new V4 version. This can be
seen in [4].
However, this new version currently contains 58 commits and is way too big
to be submitted as is. So I would like to close or discard the original V3
branch and submit V4 in pieces to make it easier to review.
Here is Part 1 of (what would be V4 of) my Builtin FSMonitor series.
Part 1 contains:
* A fix for a memory leak in the Trace2 code. (This was independently
reported in last week in "ab/tr2-leaks-and-fixes".)
* Various cleanups in the simple-ipc layer.
* A new start_bg_command() function to launch a command into the background
and wait for it to start. And a refactored consumer of it in
test-simple-ipc. There was a large discussion on commit 14/34 in V3 [5
thru 6] about the large ifdef'd blocks of platform specific code to spawn
background commands, trace2 handling, and duplicated code in
fsmonitor--daemon.c and in test-simple-ipc. That has all been addressed
here.
[1]
https://lore.kernel.org/git/pull.923.git.1617291666.gitgitgadget@gmail.com/
[2]
https://lore.kernel.org/git/pull.923.v2.git.1621691828.gitgitgadget@gmail.com/
[3]
https://lore.kernel.org/git/pull.923.v3.git.1625150864.gitgitgadget@gmail.com/
[4] https://github.com/gitgitgadget/git/pull/923 [5]
https://lore.kernel.org/git/9fe902aad87f1192705fb69ea212a2d066d0286d.1625150864.git.gitgitgadget@gmail.com/
[6] https://lore.kernel.org/git/87tukovidd.fsf@evledraar.gmail.com/
Jeff Hostetler (7):
trace2: fix memory leak of thread name
simple-ipc: preparations for supporting binary messages.
simple-ipc: move definition of ipc_active_state outside of ifdef
simple-ipc/ipc-win32: add trace2 debugging
simple-ipc/ipc-win32: add Windows ACL to named pipe
run-command: create start_bg_command
t/helper/simple-ipc: convert test-simple-ipc to use start_bg_command
compat/simple-ipc/ipc-unix-socket.c | 14 +-
compat/simple-ipc/ipc-win32.c | 176 +++++++++++++++++++--
run-command.c | 123 +++++++++++++++
run-command.h | 48 ++++++
simple-ipc.h | 21 +--
t/helper/test-simple-ipc.c | 227 ++++++++--------------------
trace2/tr2_tls.c | 1 +
7 files changed, 416 insertions(+), 194 deletions(-)
base-commit: 8b7c11b8668b4e774f81a9f0b4c30144b818f1d1
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1040%2Fjeffhostetler%2Fbuiltin-fsmonitor-part1-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1040/jeffhostetler/builtin-fsmonitor-part1-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1040
--
gitgitgadget
From: Jeff Hostetler via GitGitGadget <hidden> Date: 2021-09-15 20:36:28
From: Jeff Hostetler <redacted>
Create a variation of `run_command()` and `start_command()` to launch a command
into the background and optionally wait for it to become "ready" before returning.
Signed-off-by: Jeff Hostetler <redacted>
---
run-command.c | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++
run-command.h | 48 ++++++++++++++++++++
2 files changed, 171 insertions(+)
@@ -496,4 +496,52 @@ int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,*/voidprepare_other_repo_env(structstrvec*env_array,constchar*new_git_dir);+/**+*Possiblereturnvaluesfor`start_bg_command()`.+*/+enumstart_bg_result{+/* child process is "ready" */+SBGR_READY=0,++/* child process could not be started */+SBGR_ERROR,++/* callback error when testing for "ready" */+SBGR_CB_ERROR,++/* timeout expired waiting for child to become "ready" */+SBGR_TIMEOUT,++/* child process exited or was signalled before becomming "ready" */+SBGR_DIED,+};++/**+*Callbackusedby`start_bg_command()`toaskwhetherthe+*childprocessisreadyorneedsmoretimetobecomeready.+*+*Returns1ischildneedsmoretime(subjecttotherequestedtimeout).+*Returns0ifchildisready.+*Returns-1onanyerrorandcause`start_bg_command()`toalsoerrorout.+*/+typedefint(start_bg_wait_cb)(void*cb_data,+conststructchild_process*cmd);++/**+*Startacommandinthebackground.Waitlongenoughforthechildto+*become"ready".Captureimmediateerrors(likefailuretostart)and+*anyimmediateexitstatus(suchasashutdown/signalbeforethechild+*became"ready").+*+*Thisisacombinationof`start_command()`and`finish_command()`,but+*withacustom`wait_or_whine()`thatallowsthecallertodefinewhen+*thechildis"ready".+*+*Thecallerdoesnotneedtocall`finish_command()`.+*/+enumstart_bg_resultstart_bg_command(structchild_process*cmd,+start_bg_wait_cb*wait_cb,+void*cb_data,+unsignedinttimeout_sec);+#endif
From: Jeff Hostetler via GitGitGadget <hidden> Date: 2021-09-15 20:36:29
From: Jeff Hostetler <redacted>
Set an ACL on the named pipe to allow the well-known group EVERYONE
to read and write to the IPC server's named pipe. In the event that
the daemon was started with elevation, allow non-elevated clients
to communicate with the daemon.
Signed-off-by: Jeff Hostetler <redacted>
---
compat/simple-ipc/ipc-win32.c | 140 +++++++++++++++++++++++++++++++---
1 file changed, 129 insertions(+), 11 deletions(-)
From: Jeff Hostetler via GitGitGadget <hidden> Date: 2021-09-15 20:36:30
From: Jeff Hostetler <redacted>
Do not leak the thread name (contained within the thread context) when
a thread terminates.
Signed-off-by: Jeff Hostetler <redacted>
---
trace2/tr2_tls.c | 1 +
1 file changed, 1 insertion(+)
From: Jeff Hostetler via GitGitGadget <hidden> Date: 2021-09-15 20:36:31
From: Jeff Hostetler <redacted>
From: Carlo Marcelo Arenas Belón <redacted>
Move the declartion of the `enum ipc_active_state` type outside of
the SUPPORTS_SIMPLE_IPC ifdef.
A later commit will introduce the `fsmonitor_ipc__*()` API and stub in
a "mock" implementation that requires this enum in some function
signatures.
Signed-off-by: Carlo Marcelo Arenas Belón <redacted>
Signed-off-by: Jeff Hostetler <redacted>
---
simple-ipc.h | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
From: Jeff Hostetler via GitGitGadget <hidden> Date: 2021-09-15 20:36:33
From: Jeff Hostetler <redacted>
Add `command_len` argument to the Simple IPC API.
In my original Simple IPC API, I assumed that the request
would always be a null-terminated string of text characters.
The command arg was just a `const char *`.
I found a caller that would like to pass a binary command
to the daemon, so I want to ammend the Simple IPC API to
take `const char *command, size_t command_len` and pass
that to the daemon. (Really, the first arg should just be
a `void *` or `const unsigned byte *` to make that clearer.)
Note, the response side has always been a `struct strbuf`
which includes the buffer and length, so we already support
returning a binary answer. (Yes, it feels a little weird
returning a binary buffer in a `strbuf`, but it works.)
Signed-off-by: Jeff Hostetler <redacted>
---
compat/simple-ipc/ipc-unix-socket.c | 14 +++++++-----
compat/simple-ipc/ipc-win32.c | 14 +++++++-----
simple-ipc.h | 7 ++++--
t/helper/test-simple-ipc.c | 34 +++++++++++++++++++----------
4 files changed, 46 insertions(+), 23 deletions(-)
@@ -176,7 +177,7 @@ int ipc_client_send_command_to_connection(trace2_region_enter("ipc-client","send-command",NULL);-if(write_packetized_from_buf_no_flush(message,strlen(message),+if(write_packetized_from_buf_no_flush(message,message_len,connection->fd)<0||packet_flush_gently(connection->fd)<0){ret=error(_("could not send IPC command"));
@@ -216,7 +217,7 @@ int ipc_client_send_command_to_connection(trace2_region_enter("ipc-client","send-command",NULL);-if(write_packetized_from_buf_no_flush(message,strlen(message),+if(write_packetized_from_buf_no_flush(message,message_len,connection->fd)<0||packet_flush_gently(connection->fd)<0){ret=error(_("could not send IPC command"));
@@ -112,7 +112,7 @@ static int app__slow_command(ipc_server_reply_cb *reply_cb,/**Theclientsentacommandfollowedbya(possiblyvery)largebuffer.*/-staticintapp__sendbytes_command(constchar*received,+staticintapp__sendbytes_command(constchar*received,size_treceived_len,ipc_server_reply_cb*reply_cb,structipc_server_reply_data*reply_data){
@@ -123,6 +123,13 @@ static int app__sendbytes_command(const char *received,interrs=0;intret;+/*+*Thetestissetuptosend:+*"sendbytes"SP<n*char>+*/+if(received_len<strlen("sendbytes "))+BUG("received_len is short in app__sendbytes_command");+if(skip_prefix(received,"sendbytes ",&p))len_ballast=strlen(p);
From: Jeff Hostetler via GitGitGadget <hidden> Date: 2021-09-15 20:36:34
From: Jeff Hostetler <redacted>
Convert test helper to use `start_bg_command()` when spawning a server
daemon in the background rather than blocks of platform-specific code.
Signed-off-by: Jeff Hostetler <redacted>
---
t/helper/test-simple-ipc.c | 193 ++++++++-----------------------------
1 file changed, 40 insertions(+), 153 deletions(-)
@@ -274,178 +275,64 @@ static int daemon__run_server(void)returnret;}-#ifndef GIT_WINDOWS_NATIVE-/*-*Thisisadaptedfrom`daemonize()`.Use`fork()`todirectlycreateand-*runthedaemoninachildprocess.-*/-staticintspawn_server(pid_t*pid)-{-structipc_server_optsopts={-.nr_threads=cl_args.nr_threads,-};+staticstart_bg_wait_cbbg_wait_cb;-*pid=fork();--switch(*pid){-case0:-if(setsid()==-1)-error_errno(_("setsid failed"));-close(0);-close(1);-close(2);-sanitize_stdfds();+staticintbg_wait_cb(void*cb_data,conststructchild_process*cp)+{+ints=ipc_get_active_state(cl_args.path);-returnipc_server_run(cl_args.path,&opts,test_app_cb,-(void*)&my_app_data);+switch(s){+caseIPC_STATE__LISTENING:+/* child is "ready" */+return0;-case-1:-returnerror_errno(_("could not spawn daemon in the background"));+caseIPC_STATE__NOT_LISTENING:+caseIPC_STATE__PATH_NOT_FOUND:+/* give child more time */+return1;default:-return0;+caseIPC_STATE__INVALID_PATH:+caseIPC_STATE__OTHER_ERROR:+/* all the time in world won't help */+return-1;}}-#else-/*-*Conceptuallylike`daemonize()`butdifferentbecauseWindowsdoesnot-*have`fork(2)`.SpawnanormalWindowschildprocessbutwithoutthe-*limitationsof`start_command()`and`finish_command()`.-*/-staticintspawn_server(pid_t*pid)-{-chartest_tool_exe[MAX_PATH];-structstrvecargs=STRVEC_INIT;-intin,out;--GetModuleFileNameA(NULL,test_tool_exe,MAX_PATH);--in=open("/dev/null",O_RDONLY);-out=open("/dev/null",O_WRONLY);--strvec_push(&args,test_tool_exe);-strvec_push(&args,"simple-ipc");-strvec_push(&args,"run-daemon");-strvec_pushf(&args,"--name=%s",cl_args.path);-strvec_pushf(&args,"--threads=%d",cl_args.nr_threads);--*pid=mingw_spawnvpe(args.v[0],args.v,NULL,NULL,in,out,out);-close(in);-close(out);--strvec_clear(&args);-if(*pid<0)-returnerror(_("could not spawn daemon in the background"));--return0;-}-#endif--/*-*Thisisadaptedfrom`wait_or_whine()`.Watchthechildprocessand-*letitgetstartedandbeginlisteningforrequestsonthesocket-*beforereportingoursuccess.-*/-staticintwait_for_server_startup(pid_tpid_child)+staticintdaemon__start_server(void){-intstatus;-pid_tpid_seen;-enumipc_active_states;-time_ttime_limit,now;+structchild_processcp=CHILD_PROCESS_INIT;+enumstart_bg_resultsbgr;-time(&time_limit);-time_limit+=cl_args.max_wait_sec;+strvec_push(&cp.args,"test-tool");+strvec_push(&cp.args,"simple-ipc");+strvec_push(&cp.args,"run-daemon");+strvec_pushf(&cp.args,"--name=%s",cl_args.path);+strvec_pushf(&cp.args,"--threads=%d",cl_args.nr_threads);-for(;;){-pid_seen=waitpid(pid_child,&status,WNOHANG);+cp.no_stdin=1;+cp.no_stdout=1;+cp.no_stderr=1;-if(pid_seen==-1)-returnerror_errno(_("waitpid failed"));+sbgr=start_bg_command(&cp,bg_wait_cb,NULL,cl_args.max_wait_sec);-elseif(pid_seen==0){-/*-*Thechildisstillrunning(thisshouldbe-*thenormalcase).Trytoconnecttoiton-*thesocketandseeifitisreadyfor-*business.-*-*Ifthereisanotherdaemonalreadyrunning,-*ourchildwillfailtostart(possibly-*afteratimeoutonthelock),butwedon't-*care(whoresponds)ifthesocketislive.-*/-s=ipc_get_active_state(cl_args.path);-if(s==IPC_STATE__LISTENING)-return0;--time(&now);-if(now>time_limit)-returnerror(_("daemon not online yet"));--continue;-}+switch(sbgr){+caseSBGR_READY:+return0;-elseif(pid_seen==pid_child){-/*-*Thenewchilddaemonprocessshutdownwhile-*itwasstartingup,soitisnotlistening-*onthesocket.-*-*Trytopingthesocketintheoddchance-*thatanotherdaemonstarted(orwasalready-*running)whileourchildwasstarting.-*-*Again,wedon'tcarewhoservicesthesocket.-*/-s=ipc_get_active_state(cl_args.path);-if(s==IPC_STATE__LISTENING)-return0;+default:+caseSBGR_ERROR:+caseSBGR_CB_ERROR:+returnerror("daemon failed to start");-/*-*Wedon'tcareabouttheWEXITSTATUS()nor-*anyoftheWIF*(status)valuesbecause-*`cmd__simple_ipc()`doesthe`!!result`-*trickonallfunctionreturnvalues.-*-*Soitissufficienttojustreportthe-*earlyshutdownasanerror.-*/-returnerror(_("daemon failed to start"));-}+caseSBGR_TIMEOUT:+returnerror("daemon not online yet");-else-returnerror(_("waitpid is confused"));+caseSBGR_DIED:+returnerror("daemon terminated");}}-/*-*Thisprocesswillstartasimple-ipcserverinabackgroundprocessand-*waitforittobecomeready.Thisislike`daemonize()`butgivesus-*morecontrolandbettererrorreporting(andmakesiteasiertowrite-*unittests).-*/-staticintdaemon__start_server(void)-{-pid_tpid_child;-intret;--/*-*Runtheactualdaemoninabackgroundprocess.-*/-ret=spawn_server(&pid_child);-if(pid_child<=0)-returnret;--/*-*Lettheparentwaitforthechildprocesstogetstarted-*andbeginlisteningforrequestsonthesocket.-*/-ret=wait_for_server_startup(pid_child);--returnret;-}-/**Thisprocesswillrunaquickprobetoseeifasimple-ipcserver*isactiveonthispath.
From: Eric Sunshine <hidden> Date: 2021-09-15 20:43:47
On Wed, Sep 15, 2021 at 4:36 PM Jeff Hostetler via GitGitGadget
[off-list ref] wrote:
Add `command_len` argument to the Simple IPC API.
In my original Simple IPC API, I assumed that the request
would always be a null-terminated string of text characters.
The command arg was just a `const char *`.
I found a caller that would like to pass a binary command
to the daemon, so I want to ammend the Simple IPC API to
s/ammend/amend/
take `const char *command, size_t command_len` and pass
that to the daemon. (Really, the first arg should just be
a `void *` or `const unsigned byte *` to make that clearer.)
The reader is left wondering why you didn't also change it to `const
void *` (or one of the other choices) while at it.
Note, the response side has always been a `struct strbuf`
which includes the buffer and length, so we already support
returning a binary answer. (Yes, it feels a little weird
returning a binary buffer in a `strbuf`, but it works.)
Signed-off-by: Jeff Hostetler <redacted>
From: Taylor Blau <hidden> Date: 2021-09-16 04:53:10
On Wed, Sep 15, 2021 at 08:36:16PM +0000, Jeff Hostetler via GitGitGadget wrote:
quoted hunk
From: Jeff Hostetler <redacted>
Create a variation of `run_command()` and `start_command()` to launch a command
into the background and optionally wait for it to become "ready" before returning.
Signed-off-by: Jeff Hostetler <redacted>
---
run-command.c | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++
run-command.h | 48 ++++++++++++++++++++
2 files changed, 171 insertions(+)
This jumped out to me as unsafe, since POSIX guarantees time_t to be an
integral value holding a number of seconds (so += timeout_sec is safe
there), but it isn't in the C standard.
But we have lots of other examples of adding a number of seconds
directly the value filled in by time(2), so I think this is fine.
Small nit, probably better to write this as if (!pid_seen), but not
worth a reroll alone.
+ /*
+ * The child is currently running. Ask the callback
+ * if the child is ready to do work or whether we
+ * should keep waiting for it to boot up.
+ */
This comment is simple and informative, thank you!
+ ret = (*wait_cb)(cb_data, cmd);
+ if (!ret) {
+ /*
+ * The child is running and "ready".
+ *
+ * NEEDSWORK: As we prepare to orphan (release to
+ * the background) this child, it is not appropriate
+ * to emit a `trace2_child_exit()` event. Should we
+ * create a new event for this case?
Probably. Maybe trace2_child_orphaned() or trace2_child_background()?
+ */
+ sbgr = SBGR_READY;
+ goto done;
+ } else if (ret > 0) {
+ time_t now;
+
+ time(&now);
+ if (now < time_limit)
+ goto wait;
+
+ /*
+ * Our timeout has expired. We don't try to
+ * kill the child, but rather let it continue
+ * (hopefully) trying to startup.
+ *
+ * NEEDSWORK: Like the "ready" case, should we
+ * log a custom child-something Trace2 event here?
+ */
+ sbgr = SBGR_TIMEOUT;
+ goto done;
+ } else {
+ /*
+ * The cb gave up on this child.
+ *
+ * NEEDSWORK: Like above, should we log a custom
+ * Trace2 child-something event here?
+ */
+ sbgr = SBGR_CB_ERROR;
+ goto done;
+ }
OK, so assuming that the child is running, then we ask wait_cb what to
do. Returning zero from the callback means to background it, a positive
value means to give it more time, and negative means to cause an error.
And those match the documentation below, good.
+ if (pid_seen == cmd->pid) {
This could be an "else if", no?
+ int child_code = -1;
+
+ /*
+ * The child started, but exited or was terminated
+ * before becoming "ready".
+ *
+ * We try to match the behavior of `wait_or_whine()`
+ * and convert the child's status to a return code for
+ * tracing purposes and emit the `trace2_child_exit()`
+ * event.
+ */
+ if (WIFEXITED(wait_status))
+ child_code = WEXITSTATUS(wait_status);
+ else if (WIFSIGNALED(wait_status))
+ child_code = WTERMSIG(wait_status) + 128;
Do we care about emitting the same error (when it was signaled with
something other than SIGINT/SIGQUIT/SIGPIPE) as is reported by
wait_or_whine()?
If we want that error here, too, we could probably share the same code
here from here and in wait_or_whine(). I would probably write something
like:
static int handle_awaited_status(int status, int *code)
{
if (WIFSIGNALED(status)) {
*code = WTERMSIG(status);
if (*code != SIGINT && *code != SIGQUIT && *code != SIGPIPE)
error("%s died of signal %d", argv0, *code);
/*
* This return value is chosen so that code & 0xff
* mimics the exit code that a POSIX shell would report for
* a program that died from this signal.
*/
*code += 128;
return 1;
} else if (WIFEXITED(status)) {
*code = WEXITSTATUS(status);
return 1;
}
return 0;
}
so that we could call it in wait_or_whine() like:
} else if (!handle_awaited_status(status, &code)) {
error("waitpid is confused (%s)", argv0);
}
and similarly here in this new function. Alternatively, if we don't want
that error, then it may help future readers to add a short comment
explaining why not.
+/**
+ * Callback used by `start_bg_command()` to ask whether the
+ * child process is ready or needs more time to become ready.
+ *
+ * Returns 1 is child needs more time (subject to the requested timeout).
+ * Returns 0 if child is ready.
+ * Returns -1 on any error and cause `start_bg_command()` to also error out.
+ */
+typedef int(start_bg_wait_cb)(void *cb_data,
+ const struct child_process *cmd);
Nitpicking, but typically I would assume that the "extra" void pointer
is the last argument in a callback. It definitely does not matter,
though.
Thanks,
Taylor
From: Taylor Blau <hidden> Date: 2021-09-16 04:58:50
On Thu, Sep 16, 2021 at 12:53:07AM -0400, Taylor Blau wrote:
quoted
+/**
+ * Callback used by `start_bg_command()` to ask whether the
+ * child process is ready or needs more time to become ready.
+ *
+ * Returns 1 is child needs more time (subject to the requested timeout).
+ * Returns 0 if child is ready.
+ * Returns -1 on any error and cause `start_bg_command()` to also error out.
+ */
+typedef int(start_bg_wait_cb)(void *cb_data,
+ const struct child_process *cmd);
Nitpicking, but typically I would assume that the "extra" void pointer
is the last argument in a callback. It definitely does not matter,
though.
Looking at the last patch (which adds the first implementation of one of
these callbacks) it appears that this cb_data pointer is unused. I
assume that it is used in later patches which aren't in this topic?
If so, then it may help future readers to indicate as much in the patch
message. Perhaps "the cb_data argument in the start_bg_wait_cb callback
is unused in this series, but will be useful in later patches".
Thanks,
Taylor
From: Taylor Blau <hidden> Date: 2021-09-16 05:06:59
On Wed, Sep 15, 2021 at 08:36:17PM +0000, Jeff Hostetler via GitGitGadget wrote:
quoted hunk
From: Jeff Hostetler <redacted>
Convert test helper to use `start_bg_command()` when spawning a server
daemon in the background rather than blocks of platform-specific code.
Signed-off-by: Jeff Hostetler <redacted>
---
t/helper/test-simple-ipc.c | 193 ++++++++-----------------------------
1 file changed, 40 insertions(+), 153 deletions(-)
@@ -274,178 +275,64 @@ static int daemon__run_server(void)returnret;}-#ifndef GIT_WINDOWS_NATIVE-/*-*Thisisadaptedfrom`daemonize()`.Use`fork()`todirectlycreateand-*runthedaemoninachildprocess.-*/-staticintspawn_server(pid_t*pid)-{-structipc_server_optsopts={-.nr_threads=cl_args.nr_threads,-};+staticstart_bg_wait_cbbg_wait_cb;
This whole patch is delightful to read, as the new implementation is so
much cleaner as a result of the earlier work in this series.
Am I correct in assuming that this is to encourage a compiler error if
bg_wait_cb does not satisfy the type of start_bg_wait_cb? If so, then I
think we are already getting that by trying to pass bg_wait_cb to
start_bg_command().
E.g., applying this (intentionally broken) diff on top:
@@ -275,9 +275,7 @@ static int daemon__run_server(void)returnret;}-staticstart_bg_wait_cbbg_wait_cb;--staticintbg_wait_cb(void*cb_data,conststructchild_process*cp)+staticintbg_wait_cb(constvoid*cb_data,conststructchild_process*cp){ints=ipc_get_active_state(cl_args.path);--->8---
and then compiling still warns of a mismatched type when calling
start_bg_command().
- *pid = fork();
-
- switch (*pid) {
- case 0:
- if (setsid() == -1)
- error_errno(_("setsid failed"));
- close(0);
- close(1);
- close(2);
- sanitize_stdfds();
+static int bg_wait_cb(void *cb_data, const struct child_process *cp)
+{
+ int s = ipc_get_active_state(cl_args.path);
- return ipc_server_run(cl_args.path, &opts, test_app_cb,
- (void*)&my_app_data);
+ switch (s) {
+ case IPC_STATE__LISTENING:
+ /* child is "ready" */
+ return 0;
- case -1:
- return error_errno(_("could not spawn daemon in the background"));
+ case IPC_STATE__NOT_LISTENING:
+ case IPC_STATE__PATH_NOT_FOUND:
+ /* give child more time */
+ return 1;
default:
I'm always a little hesitant to have default cases when switch over enum
types, since it suppresses the warning when there's a new value of that
type. But we already have a similar default in client__probe_server().
- else if (pid_seen == pid_child) {
- /*
- * The new child daemon process shutdown while
- * it was starting up, so it is not listening
- * on the socket.
- *
- * Try to ping the socket in the odd chance
- * that another daemon started (or was already
- * running) while our child was starting.
- *
- * Again, we don't care who services the socket.
- */
- s = ipc_get_active_state(cl_args.path);
- if (s == IPC_STATE__LISTENING)
- return 0;
+ default:
On Wed, Sep 15 2021, Jeff Hostetler via GitGitGadget wrote:
quoted hunk
From: Jeff Hostetler <redacted>
Do not leak the thread name (contained within the thread context) when
a thread terminates.
Signed-off-by: Jeff Hostetler <redacted>
---
trace2/tr2_tls.c | 1 +
1 file changed, 1 insertion(+)
As noted in your cover letter:
* A fix for a memory leak in the Trace2 code. (This was independently
reported in last week in "ab/tr2-leaks-and-fixes".)
So I think this patch can be dropped from this series, since it's exact
duplicate of my 48f68715b14 (tr2: stop leaking "thread_name" memory,
2021-08-27) in ab/tr2-leaks-and-fixes, currently in "next" and marked
for a merge with master.
When submitting a series that depends on another one it's best to rebase
it on top of it & indicate it as such in the cover letter, Junio can
queue such a series on top of another one.
In this case I'm still not sure why this fix is here, i.e. surely
nothing later in the series absolutely needs this stray memory leak
fix...
From: Taylor Blau <hidden> Date: 2021-09-16 05:43:14
On Thu, Sep 16, 2021 at 07:35:59AM +0200, Ævar Arnfjörð Bjarmason wrote:
So I think this patch can be dropped from this series, since it's exact
duplicate of my 48f68715b14 (tr2: stop leaking "thread_name" memory,
2021-08-27) in ab/tr2-leaks-and-fixes, currently in "next" and marked
for a merge with master.
I agree it can be dropped.
When submitting a series that depends on another one it's best to rebase
it on top of it & indicate it as such in the cover letter, Junio can
queue such a series on top of another one.
In this case I'm still not sure why this fix is here, i.e. surely
nothing later in the series absolutely needs this stray memory leak
fix...
But there's no need for Jeff to depend on your branch, since (as you
mentioned) this cleanup isn't relevant for anything else in this series,
which is a sort of grab-bag of miscellaneous clean-ups.
Thanks,
Taylor
I've never used this Win32 API (or well, any Win32 API) but I'm guessing
that GetLastError() isn't here to check an error in GetLastError() itself.
Earlier in this function added in your 59c7b88198a (simple-ipc: add
win32 implementation, 2021-03-15) we assign to "gle", I'd really expect...:
...something that looks exactly like this. I.e. as shown by the below
hunk-at-the-end, as it is I'm either missing some subtlety that could
really use explaining. I.e. this reads like:
int saved_errno = errno;
if (syscall()) {
if (errno)
die("bad");
saved_errno = errno;
log_it("...%d", saved_errno);
}
When surely we want either of:
int saved_errno = errno;
if (syscall()) {
if (errno)
die("bad");
log_it("...%d", errno);
}
Or better yet (and consistent with the rest of your code):
int saved_errno = errno;
if (syscall()) {
saved_errno = errno;
if (saved_errno)
die("bad");
log_it("...%d", saved_errno);
}
Just a nit on the init pattern, since we always allocate this on the
stack (this as all the relevant "struct my_sa_data") I'd have thought to
see:
#define INIT_MY_SA_DATA { 0 }
[...]
struct my_sa_data my_sa_data = INIT_MY_SA_DATA;
Which gets rid of the need for an init_sa() function.
Also having the release_sa() do a memset() is a bit odd, usually we have
a reset*() function do that if the intent is to re-use, but it doesn't
appear to be in this case, and we don't return this data anywhere, do
we?
On Wed, Sep 15 2021, Jeff Hostetler via GitGitGadget wrote:
[...]
+ default:
+ case SBGR_ERROR:
+ case SBGR_CB_ERROR:
+ return error("daemon failed to start");
- /*
- * We don't care about the WEXITSTATUS() nor
- * any of the WIF*(status) values because
- * `cmd__simple_ipc()` does the `!!result`
- * trick on all function return values.
- *
- * So it is sufficient to just report the
- * early shutdown as an error.
- */
- return error(_("daemon failed to start"));
- }
+ case SBGR_TIMEOUT:
+ return error("daemon not online yet");
- else
- return error(_("waitpid is confused"));
+ case SBGR_DIED:
+ return error("daemon terminated");
}
}
It's not mentioned in the commit message, but the while-we're-at-it
dropping of _() makes sense here, it shouldn't have been used in a test
helper to begin with. I.e. translators don't need to be translating
stuff purely internal to the test suite.
On Wed, Sep 15 2021, Jeff Hostetler via GitGitGadget wrote:
quoted hunk
From: Jeff Hostetler <redacted>
Create a variation of `run_command()` and `start_command()` to launch a command
into the background and optionally wait for it to become "ready" before returning.
Signed-off-by: Jeff Hostetler <redacted>
---
run-command.c | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++
run-command.h | 48 ++++++++++++++++++++
2 files changed, 171 insertions(+)
Since we have no existing users, can we change this to:
if (cmd->clean_on_exit)
BUG("start_bg_command() doesn't support non-zero clean_on_exit");
Just silently discarding what the caller asked for seems like the wrong
thing to do, why the silence?
+
+ ret = start_command(cmd);
+ if (ret) {
+ /*
+ * We assume that if `start_command()` fails, we
+ * either get a complete `trace2_child_start() /
+ * trace2_child_exit()` pair or it fails before the
+ * `trace2_child_start()` is emitted, so we do not
+ * need to worry about it here.
+ *
+ * We also assume that `start_command()` does not add
+ * us to the cleanup list. And that it calls
+ * calls `child_process_clear()`.
+ */
These all look like sensible things to assume, but I think commentary /
writing on this would be much better just documenting that the
start_command() API does this in its comment in run-command.h, or
perhaps the comment starting with "The functions: child_process_init".
+ sbgr = SBGR_ERROR;
+ goto done;
+ }
+
+ time(&time_limit);
+ time_limit += timeout_sec;
+
+wait:
+ pid_seen = waitpid(cmd->pid, &wait_status, WNOHANG);
+
+ if (pid_seen == 0) {
+ /*
+ * The child is currently running. Ask the callback
+ * if the child is ready to do work or whether we
+ * should keep waiting for it to boot up.
+ */
+ ret = (*wait_cb)(cb_data, cmd);
+ if (!ret) {
+ /*
+ * The child is running and "ready".
+ *
+ * NEEDSWORK: As we prepare to orphan (release to
+ * the background) this child, it is not appropriate
+ * to emit a `trace2_child_exit()` event. Should we
+ * create a new event for this case?
+ */
+ sbgr = SBGR_READY;
+ goto done;
Per api-trace2.txt:
`"child_exit"`::
This event is generated after the current process has returned
from the waitpid() and collected the exit information from the
child.
My (perhaps wrong) reading of that is that yes, we should do that, after
all if we've released the child are we otherwise going to hear from them
again in any way trace2 could log with a child_exit later?
Ditto the "commentary better elsewhere" whatever the result of this
discussion is, let's add a note to api-trace2.txt about how detached
children are handled by child_exit events.
[...]
+ * NEEDSWORK: Like the "ready" case, should we
+ * log a custom child-something Trace2 event here?
+ */
+ sbgr = SBGR_TIMEOUT;
+ goto done;
[...]
+ } else {
+ /*
+ * The cb gave up on this child.
+ *
+ * NEEDSWORK: Like above, should we log a custom
+ * Trace2 child-something event here?
+ */
+ sbgr = SBGR_CB_ERROR;
+ goto done;
*ditto*
+ }
+ }
+
+ if (pid_seen == cmd->pid) {
+ int child_code = -1;
+
+ /*
+ * The child started, but exited or was terminated
+ * before becoming "ready".
+ *
+ * We try to match the behavior of `wait_or_whine()`
+ * and convert the child's status to a return code for
+ * tracing purposes and emit the `trace2_child_exit()`
+ * event.
+ */
+ if (WIFEXITED(wait_status))
+ child_code = WEXITSTATUS(wait_status);
+ else if (WIFSIGNALED(wait_status))
+ child_code = WTERMSIG(wait_status) + 128;
+ trace2_child_exit(cmd, child_code);
Although with the above: Perhaps I've missed some subtleties...
@@ -496,4 +496,52 @@ int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,*/voidprepare_other_repo_env(structstrvec*env_array,constchar*new_git_dir);+/**+*Possiblereturnvaluesfor`start_bg_command()`.+*/+enumstart_bg_result{+/* child process is "ready" */+SBGR_READY=0,
Clarity nit, whenever I see an explicit enum assignment I start hunting
for things that depend on the specific value, as we sometimes do, but
there appears to be nothing...
+ /* child process could not be started */
+ SBGR_ERROR,
+
+ /* callback error when testing for "ready" */
+ SBGR_CB_ERROR,
+
+ /* timeout expired waiting for child to become "ready" */
+ SBGR_TIMEOUT,
+
+ /* child process exited or was signalled before becomming "ready" */
+ SBGR_DIED,
...I'd think if we were tweaking the value we'd want to put all the
error values at < 0, but I'm fine with this pattern of them being
positive, it encourages users to use switch/case & compiler checks, and
since it's all new users...
+ * This is a combination of `start_command()` and `finish_command()`, but
Doc nit: I think with whatever syntax standard we use for /** comments
that we prefer functions like_this() not quoted `like_this()`, that
being for values like `123` or whatever. But I'm just going by a quick
grep there & what Emacs is highlighting.
On Thu, Sep 16, 2021 at 07:35:59AM +0200, Ævar Arnfjörð Bjarmason wrote:
quoted
So I think this patch can be dropped from this series, since it's exact
duplicate of my 48f68715b14 (tr2: stop leaking "thread_name" memory,
2021-08-27) in ab/tr2-leaks-and-fixes, currently in "next" and marked
for a merge with master.
I agree it can be dropped.
quoted
When submitting a series that depends on another one it's best to rebase
it on top of it & indicate it as such in the cover letter, Junio can
queue such a series on top of another one.
In this case I'm still not sure why this fix is here, i.e. surely
nothing later in the series absolutely needs this stray memory leak
fix...
But there's no need for Jeff to depend on your branch, since (as you
mentioned) this cleanup isn't relevant for anything else in this series,
which is a sort of grab-bag of miscellaneous clean-ups.
Indeed, to be clear it was just general advice about queue-on-top.
But to clarify what I was getting at here: If we just came up with the
same diff I'd have assumed Jeff just hadn't need the change in "next",
but since he clearly has I was confused by it being here.
I.e. it doesn't *seem* like anything in the rest of the series depends
on it, so why have it here at all since the bug is being fixed anyway?
Or if it does depend on it in some subtle way I've missed, perhaps it
does need to be queued on top of ab/tr2-leaks-and-fixes, and the
relevant commit/subtle dependency needs to be called out in a commit
message.
Or maybe Jeff had just come up with this independently, noticed it just
before submission and just updated the CL, not the patch or series
itself :)
From: Jeff Hostetler <hidden> Date: 2021-09-16 15:35:56
On 9/16/21 4:01 AM, Ævar Arnfjörð Bjarmason wrote:
On Thu, Sep 16 2021, Taylor Blau wrote:
quoted
On Thu, Sep 16, 2021 at 07:35:59AM +0200, Ævar Arnfjörð Bjarmason wrote:
quoted
So I think this patch can be dropped from this series, since it's exact
duplicate of my 48f68715b14 (tr2: stop leaking "thread_name" memory,
2021-08-27) in ab/tr2-leaks-and-fixes, currently in "next" and marked
for a merge with master.
I agree it can be dropped.
quoted
When submitting a series that depends on another one it's best to rebase
it on top of it & indicate it as such in the cover letter, Junio can
queue such a series on top of another one.
In this case I'm still not sure why this fix is here, i.e. surely
nothing later in the series absolutely needs this stray memory leak
fix...
But there's no need for Jeff to depend on your branch, since (as you
mentioned) this cleanup isn't relevant for anything else in this series,
which is a sort of grab-bag of miscellaneous clean-ups.
Indeed, to be clear it was just general advice about queue-on-top.
But to clarify what I was getting at here: If we just came up with the
same diff I'd have assumed Jeff just hadn't need the change in "next",
but since he clearly has I was confused by it being here.
I.e. it doesn't *seem* like anything in the rest of the series depends
on it, so why have it here at all since the bug is being fixed anyway?
Or if it does depend on it in some subtle way I've missed, perhaps it
does need to be queued on top of ab/tr2-leaks-and-fixes, and the
relevant commit/subtle dependency needs to be called out in a commit
message.
Or maybe Jeff had just come up with this independently, noticed it just
before submission and just updated the CL, not the patch or series
itself :)
I'll drop this commit since your version is already queued up
and headed to master. I've been carrying it in my dev branch
for a while and was using it to make leak reporting a little
quieter.
And yes, I just noticed that yours had advanced when I wrote the
cover letter and ACKd it rather than dropping it.
And no, nothing in the rest of the whole FSMonitor series depends
on this, so I can leave my series based upon master rather than
your branch.
Thanks
Jeff
On 9/16/21 4:01 AM, Ævar Arnfjörð Bjarmason wrote:
quoted
On Thu, Sep 16 2021, Taylor Blau wrote:
quoted
On Thu, Sep 16, 2021 at 07:35:59AM +0200, Ævar Arnfjörð Bjarmason wrote:
quoted
So I think this patch can be dropped from this series, since it's exact
duplicate of my 48f68715b14 (tr2: stop leaking "thread_name" memory,
2021-08-27) in ab/tr2-leaks-and-fixes, currently in "next" and marked
for a merge with master.
I agree it can be dropped.
quoted
When submitting a series that depends on another one it's best to rebase
it on top of it & indicate it as such in the cover letter, Junio can
queue such a series on top of another one.
In this case I'm still not sure why this fix is here, i.e. surely
nothing later in the series absolutely needs this stray memory leak
fix...
But there's no need for Jeff to depend on your branch, since (as you
mentioned) this cleanup isn't relevant for anything else in this series,
which is a sort of grab-bag of miscellaneous clean-ups.
Indeed, to be clear it was just general advice about queue-on-top.
But to clarify what I was getting at here: If we just came up with
the
same diff I'd have assumed Jeff just hadn't need the change in "next",
but since he clearly has I was confused by it being here.
I.e. it doesn't *seem* like anything in the rest of the series
depends
on it, so why have it here at all since the bug is being fixed anyway?
Or if it does depend on it in some subtle way I've missed, perhaps it
does need to be queued on top of ab/tr2-leaks-and-fixes, and the
relevant commit/subtle dependency needs to be called out in a commit
message.
Or maybe Jeff had just come up with this independently, noticed it
just
before submission and just updated the CL, not the patch or series
itself :)
I'll drop this commit since your version is already queued up
and headed to master. I've been carrying it in my dev branch
for a while and was using it to make leak reporting a little
quieter.
And yes, I just noticed that yours had advanced when I wrote the
cover letter and ACKd it rather than dropping it.
And no, nothing in the rest of the whole FSMonitor series depends
on this, so I can leave my series based upon master rather than
your branch.
Thanks
Jeff
Sounds good, thanks for clarifying.
In any case by the time you'll re-roll this (or soon thereafter) Junio
will probably have merged it down anyway.
From: Jeff Hostetler <hidden> Date: 2021-09-17 16:52:31
On 9/15/21 4:43 PM, Eric Sunshine wrote:
On Wed, Sep 15, 2021 at 4:36 PM Jeff Hostetler via GitGitGadget
[off-list ref] wrote:
quoted
Add `command_len` argument to the Simple IPC API.
In my original Simple IPC API, I assumed that the request
would always be a null-terminated string of text characters.
The command arg was just a `const char *`.
I found a caller that would like to pass a binary command
to the daemon, so I want to ammend the Simple IPC API to
s/ammend/amend/
quoted
take `const char *command, size_t command_len` and pass
that to the daemon. (Really, the first arg should just be
a `void *` or `const unsigned byte *` to make that clearer.)
The reader is left wondering why you didn't also change it to `const
void *` (or one of the other choices) while at it.
The simple ipc layer just passes the buffer to the pkt-line layer
and it takes a "const char *", so to avoid confusion I just left
the type is it was. If later we want to fix pkt-line, we can
investigate passing a "const unsigned byte *" value down the
call chain, but that is more than I want to do right now.
quoted
Note, the response side has always been a `struct strbuf`
which includes the buffer and length, so we already support
returning a binary answer. (Yes, it feels a little weird
returning a binary buffer in a `strbuf`, but it works.)
Signed-off-by: Jeff Hostetler <redacted>
@@ -109,9 +109,12 @@ static enum ipc_active_state connect_to_server( t_start_ms = (DWORD)(getnanotime() / 1000000); if (!WaitNamedPipeW(wpath, timeout_ms)) {- if (GetLastError() == ERROR_SEM_TIMEOUT)+ gle = GetLastError();+ if (gle == ERROR_SEM_TIMEOUT) return IPC_STATE__NOT_LISTENING;+ /* ...rest of your patch */+ return IPC_STATE__OTHER_ERROR; }
Yeah, I was just trying to minimize the size of the diff
and at the time was considering those debug messages to be
temporary, but I kind of like having them so it makes sense
to clean up a bit as you've indicated.
Thanks
Jeff
Just a nit on the init pattern, since we always allocate this on the
stack (this as all the relevant "struct my_sa_data") I'd have thought to
see:
#define INIT_MY_SA_DATA { 0 }
[...]
struct my_sa_data my_sa_data = INIT_MY_SA_DATA;
Which gets rid of the need for an init_sa() function.
The current "my_sa_data" is just a set of 4 pointers, so yes your
INIT_MY_SA_DATA macro and my init_sa function are equivalent.
(And assuming that mine and memset are inlined by the compiler, they
are both probably exactly equivalent.) So it really doesn't matter
one way or the other.
Also having the release_sa() do a memset() is a bit odd, usually we have
a reset*() function do that if the intent is to re-use, but it doesn't
appear to be in this case, and we don't return this data anywhere, do
we?
I use the release_sa() function inside my "fail:" label in get_sa()
to cleanup if any of the component parts of the SA cannot be created.
So the return value of get_sa() is either fully constructed or contains
NULL pointers.
(The docs are jargon heavy and little obscure and circular and it
is not at all clear if/when we might fail to create some of these
sub-structures.)
So I allow the SA failure to be silently ignored and we create the named
pipe without an ACL. Normally, this is fine since the daemon usually
isn't elevated.
In create_new_pipe we always call release_sa to free the pointers if
necessary. (So in case of a failure in get_sa, we won't double free
the pointers when create_new_pipe calls release_sa.)
Another way to think of it is that in release_sa, I'd like to have
FREE_AND_NULL() variants for FreeSid() and LocalFree(), but a simple
memset is sufficient.
Jeff
Just a nit on the init pattern, since we always allocate this on the
stack (this as all the relevant "struct my_sa_data") I'd have thought to
see:
#define INIT_MY_SA_DATA { 0 }
[...]
struct my_sa_data my_sa_data = INIT_MY_SA_DATA;
Which gets rid of the need for an init_sa() function.
The current "my_sa_data" is just a set of 4 pointers, so yes your
INIT_MY_SA_DATA macro and my init_sa function are equivalent.
(And assuming that mine and memset are inlined by the compiler, they
are both probably exactly equivalent.) So it really doesn't matter
one way or the other.
Yes it's the same to the compiler, see 5726a6b4012 (*.c *_init(): define
in terms of corresponding *_INIT macro, 2021-07-01).
But being the same to the compiler != the same to human readers. I was
going for the latter here, i.e. in git.git init with a macro tends to be
used if the init is simple enough to be run like that, and with a
function if it really does need to run some function (and then after
5726a6b4012 the init-via-function-via-macro for things that need init
after malloc() or whatever).
Whereas this one's just on the stack, and doesn't need anything special,
so the nit was about just preferring the simplest construct for the job.
quoted
Also having the release_sa() do a memset() is a bit odd, usually we
have
a reset*() function do that if the intent is to re-use, but it doesn't
appear to be in this case, and we don't return this data anywhere, do
we?
I use the release_sa() function inside my "fail:" label in get_sa()
to cleanup if any of the component parts of the SA cannot be created.
So the return value of get_sa() is either fully constructed or contains
NULL pointers.
(The docs are jargon heavy and little obscure and circular and it
is not at all clear if/when we might fail to create some of these
sub-structures.)
So I allow the SA failure to be silently ignored and we create the named
pipe without an ACL. Normally, this is fine since the daemon usually
isn't elevated.
In create_new_pipe we always call release_sa to free the pointers if
necessary. (So in case of a failure in get_sa, we won't double free
the pointers when create_new_pipe calls release_sa.)
Another way to think of it is that in release_sa, I'd like to have
FREE_AND_NULL() variants for FreeSid() and LocalFree(), but a simple
memset is sufficient.
*nod*, I meant if functions are doing that sort of intent-to-reuse we
usually call them *_reset(), with a *_release() that's just a plain
free() and forget.
From: Jeff Hostetler <hidden> Date: 2021-09-17 19:41:57
On 9/16/21 1:06 AM, Taylor Blau wrote:
On Wed, Sep 15, 2021 at 08:36:17PM +0000, Jeff Hostetler via GitGitGadget wrote:
quoted
From: Jeff Hostetler <redacted>
Convert test helper to use `start_bg_command()` when spawning a server
daemon in the background rather than blocks of platform-specific code.
Signed-off-by: Jeff Hostetler <redacted>
---
t/helper/test-simple-ipc.c | 193 ++++++++-----------------------------
1 file changed, 40 insertions(+), 153 deletions(-)
@@ -274,178 +275,64 @@ static int daemon__run_server(void)returnret;}-#ifndef GIT_WINDOWS_NATIVE-/*-*Thisisadaptedfrom`daemonize()`.Use`fork()`todirectlycreateand-*runthedaemoninachildprocess.-*/-staticintspawn_server(pid_t*pid)-{-structipc_server_optsopts={-.nr_threads=cl_args.nr_threads,-};+staticstart_bg_wait_cbbg_wait_cb;
This whole patch is delightful to read, as the new implementation is so
much cleaner as a result of the earlier work in this series.
Am I correct in assuming that this is to encourage a compiler error if
bg_wait_cb does not satisfy the type of start_bg_wait_cb? If so, then I
think we are already getting that by trying to pass bg_wait_cb to
start_bg_command().
I use that trick to get the compiler to give me a compiler error at the
point of the function declaration.
For example, If I add an arg to the function that doesn't match what's
in the prototype definition, I get:
t/helper/test-simple-ipc.c:280:12: error: conflicting types for 'bg_wait_cb'
static int bg_wait_cb(const struct child_process *cp, void *cb_data, int
foo)
^
t/helper/test-simple-ipc.c:278:25: note: previous declaration is here
static start_bg_wait_cb bg_wait_cb;
^
1 error generated.
Yes, we may get an error when the function pointer is referenced in
start_bg_command() or if we're using it to initialize a vtable or
something, but those errors are further away from the actual error
(and sometimes they can be a little cryptic).
Also, it helps document that this function's signature is predefined
for a reason.
It's a quirky trick I know, but it has served me well over the years.
quoted hunk
E.g., applying this (intentionally broken) diff on top:
@@ -275,9 +275,7 @@ static int daemon__run_server(void)returnret;}-staticstart_bg_wait_cbbg_wait_cb;--staticintbg_wait_cb(void*cb_data,conststructchild_process*cp)+staticintbg_wait_cb(constvoid*cb_data,conststructchild_process*cp){ints=ipc_get_active_state(cl_args.path);--->8---
and then compiling still warns of a mismatched type when calling
start_bg_command().
quoted
- *pid = fork();
-
- switch (*pid) {
- case 0:
- if (setsid() == -1)
- error_errno(_("setsid failed"));
- close(0);
- close(1);
- close(2);
- sanitize_stdfds();
+static int bg_wait_cb(void *cb_data, const struct child_process *cp)
+{
+ int s = ipc_get_active_state(cl_args.path);
- return ipc_server_run(cl_args.path, &opts, test_app_cb,
- (void*)&my_app_data);
+ switch (s) {
+ case IPC_STATE__LISTENING:
+ /* child is "ready" */
+ return 0;
- case -1:
- return error_errno(_("could not spawn daemon in the background"));
+ case IPC_STATE__NOT_LISTENING:
+ case IPC_STATE__PATH_NOT_FOUND:
+ /* give child more time */
+ return 1;
default:
I'm always a little hesitant to have default cases when switch over enum
types, since it suppresses the warning when there's a new value of that
type. But we already have a similar default in client__probe_server().
Do all compilers now handle switching over an enum and detect unhandled
cases? Once upon a time that wasn't the case IIRC.
On Wed, Sep 15, 2021 at 08:36:17PM +0000, Jeff Hostetler via GitGitGadget wrote:
quoted
From: Jeff Hostetler <redacted>
Convert test helper to use `start_bg_command()` when spawning a server
daemon in the background rather than blocks of platform-specific code.
Signed-off-by: Jeff Hostetler <redacted>
---
t/helper/test-simple-ipc.c | 193 ++++++++-----------------------------
1 file changed, 40 insertions(+), 153 deletions(-)
@@ -274,178 +275,64 @@ static int daemon__run_server(void)returnret;}-#ifndef GIT_WINDOWS_NATIVE-/*-*Thisisadaptedfrom`daemonize()`.Use`fork()`todirectlycreateand-*runthedaemoninachildprocess.-*/-staticintspawn_server(pid_t*pid)-{-structipc_server_optsopts={-.nr_threads=cl_args.nr_threads,-};+staticstart_bg_wait_cbbg_wait_cb;
This whole patch is delightful to read, as the new implementation is
so
much cleaner as a result of the earlier work in this series.
Am I correct in assuming that this is to encourage a compiler error
if
bg_wait_cb does not satisfy the type of start_bg_wait_cb? If so, then I
think we are already getting that by trying to pass bg_wait_cb to
start_bg_command().
I use that trick to get the compiler to give me a compiler error at the
point of the function declaration.
For example, If I add an arg to the function that doesn't match what's
in the prototype definition, I get:
t/helper/test-simple-ipc.c:280:12: error: conflicting types for 'bg_wait_cb'
static int bg_wait_cb(const struct child_process *cp, void *cb_data,
int foo)
^
t/helper/test-simple-ipc.c:278:25: note: previous declaration is here
static start_bg_wait_cb bg_wait_cb;
^
1 error generated.
Yes, we may get an error when the function pointer is referenced in
start_bg_command() or if we're using it to initialize a vtable or
something, but those errors are further away from the actual error
(and sometimes they can be a little cryptic).
Also, it helps document that this function's signature is predefined
for a reason.
It's a quirky trick I know, but it has served me well over the years.
I haven't seen this idiom before. I think it's best to avoid patterns
designed to massage messages out of any specific compilers/versions.
It seems inevitable that it'll either be counter-productive or
redundant. Here with clang v11 doing this makes the warning
worse. I.e. without the forward declaration:
t/helper/test-simple-ipc.c:315:31: error: incompatible function
pointer types passing 'int (void *, const struct child_process *,
int)' to parameter of type ' start_bg_wait_cb *' (aka 'int (*)(void
*, const struct child_process *)')
[-Werror,-Wincompatible-function-pointer-types]
sbgr = start_bg_command(&cp, bg_wait_cb, NULL, cl_args.max_wait_sec);
^~~~~~~~~~
./run-command.h:564:29: note: passing argument to parameter 'wait_cb' here
start_bg_wait_cb *wait_cb,
^
1 error generated.
I.e. we get the specific warning category for this type of error
(-Werror,-Wincompatible-function-pointer-types), and we're pointed at
the caller in question (which to be fair, it seems you don't prefer),
but also a reference to the run-command.h definition.
Most importantly, we get quoted what the type is/should be, which is
missing with the forward declaration. It's the equivalent of saying "you
did bad!" instead of "you did bad X, do Y instead!".
quoted
E.g., applying this (intentionally broken) diff on top:
@@ -275,9 +275,7 @@ static int daemon__run_server(void)returnret;}-staticstart_bg_wait_cbbg_wait_cb;--staticintbg_wait_cb(void*cb_data,conststructchild_process*cp)+staticintbg_wait_cb(constvoid*cb_data,conststructchild_process*cp){ints=ipc_get_active_state(cl_args.path);--->8---
and then compiling still warns of a mismatched type when calling
start_bg_command().
quoted
- *pid = fork();
-
- switch (*pid) {
- case 0:
- if (setsid() == -1)
- error_errno(_("setsid failed"));
- close(0);
- close(1);
- close(2);
- sanitize_stdfds();
+static int bg_wait_cb(void *cb_data, const struct child_process *cp)
+{
+ int s = ipc_get_active_state(cl_args.path);
- return ipc_server_run(cl_args.path, &opts, test_app_cb,
- (void*)&my_app_data);
+ switch (s) {
+ case IPC_STATE__LISTENING:
+ /* child is "ready" */
+ return 0;
- case -1:
- return error_errno(_("could not spawn daemon in the background"));
+ case IPC_STATE__NOT_LISTENING:
+ case IPC_STATE__PATH_NOT_FOUND:
+ /* give child more time */
+ return 1;
default:
I'm always a little hesitant to have default cases when switch over
enum
types, since it suppresses the warning when there's a new value of that
type. But we already have a similar default in client__probe_server().
Do all compilers now handle switching over an enum and detect unhandled
cases? Once upon a time that wasn't the case IIRC.
I don't think so, but the ones we widely use do, i.e. clang and gcc at
least.
For this sort of thing it really doesn't matter if *all* compilers
support it, since we'll only need to catch such "missing enum arm"
issues with one of them.
E.g. in my 338abb0f045 (builtins + test helpers: use return instead of
exit() in cmd_*, 2021-06-08) I fixed something that I've only gotten
Oracle SunCC to emit (gcc and clang don't detect it), but as long as
that one compiler does & someone checks it regularly...
By having a "default" case you're hiding that detection from the
compilers capable of detecting a logic error in this code, whereas if
the compiler can't do that it'll just ignore it.
From: Jeff Hostetler via GitGitGadget <hidden> Date: 2021-09-20 15:36:24
Here is V2 of Part 1 of my Builtin FSMonitor series.
Changes since V1 include:
* Drop the Trace2 memory leak.
* Added a new "child_ready" event to Trace2 as an alternative to the
"child_exit" event for background processes.
* Convert the Trace2-related NEEDSWORK items in "start_bg_command()" to use
the new "child_ready" event.
* Various minor code and documentation cleanups.
Jeff Hostetler (7):
trace2: add trace2_child_ready() to report on background children
simple-ipc: preparations for supporting binary messages.
simple-ipc: move definition of ipc_active_state outside of ifdef
simple-ipc/ipc-win32: add trace2 debugging
simple-ipc/ipc-win32: add Windows ACL to named pipe
run-command: create start_bg_command
t/helper/simple-ipc: convert test-simple-ipc to use start_bg_command
Documentation/technical/api-trace2.txt | 40 +++++
compat/simple-ipc/ipc-unix-socket.c | 14 +-
compat/simple-ipc/ipc-win32.c | 179 +++++++++++++++++--
run-command.c | 129 ++++++++++++++
run-command.h | 57 ++++++
simple-ipc.h | 21 ++-
t/helper/test-simple-ipc.c | 233 +++++++------------------
trace2.c | 31 ++++
trace2.h | 25 +++
trace2/tr2_tgt.h | 5 +
trace2/tr2_tgt_event.c | 22 +++
trace2/tr2_tgt_normal.c | 14 ++
trace2/tr2_tgt_perf.c | 15 ++
13 files changed, 587 insertions(+), 198 deletions(-)
base-commit: 8b7c11b8668b4e774f81a9f0b4c30144b818f1d1
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1040%2Fjeffhostetler%2Fbuiltin-fsmonitor-part1-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1040/jeffhostetler/builtin-fsmonitor-part1-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/1040
Range-diff vs v1:
1: 5f557caee00 < -: ----------- trace2: fix memory leak of thread name
-: ----------- > 1: f88e9feff26 trace2: add trace2_child_ready() to report on background children
2: 7182419f6df ! 2: 258baa0df8c simple-ipc: preparations for supporting binary messages.
@@ Commit message
Add `command_len` argument to the Simple IPC API.
- In my original Simple IPC API, I assumed that the request
- would always be a null-terminated string of text characters.
- The command arg was just a `const char *`.
+ In my original Simple IPC API, I assumed that the request would always
+ be a null-terminated string of text characters. The `command`
+ argument was just a `const char *`.
- I found a caller that would like to pass a binary command
- to the daemon, so I want to ammend the Simple IPC API to
- take `const char *command, size_t command_len` and pass
- that to the daemon. (Really, the first arg should just be
- a `void *` or `const unsigned byte *` to make that clearer.)
+ I found a caller that would like to pass a binary command to the
+ daemon, so I am amending the Simple IPC API to receive `const char
+ *command, size_t command_len` arguments.
- Note, the response side has always been a `struct strbuf`
- which includes the buffer and length, so we already support
- returning a binary answer. (Yes, it feels a little weird
- returning a binary buffer in a `strbuf`, but it works.)
+ I considered changing the `command` argument to be a `void *`, but the
+ IPC layer simply passes it to the pkt-line layer which takes a `const
+ char *`, so to avoid confusion I left it as is.
+
+ Note, the response side has always been a `struct strbuf` which
+ includes the buffer and length, so we already support returning a
+ binary answer. (Yes, it feels a little weird returning a binary
+ buffer in a `strbuf`, but it works.)
Signed-off-by: Jeff Hostetler [off-list ref]
3: 7de207828ca = 3: c94b4cbcbf2 simple-ipc: move definition of ipc_active_state outside of ifdef
4: 30b7bb247c3 ! 4: 82b6ce0dd6a simple-ipc/ipc-win32: add trace2 debugging
@@ compat/simple-ipc/ipc-win32.c: static enum ipc_active_state get_active_state(wch
}
@@ compat/simple-ipc/ipc-win32.c: static enum ipc_active_state connect_to_server(
- if (GetLastError() == ERROR_SEM_TIMEOUT)
+ t_start_ms = (DWORD)(getnanotime() / 1000000);
+
+ if (!WaitNamedPipeW(wpath, timeout_ms)) {
+- if (GetLastError() == ERROR_SEM_TIMEOUT)
++ DWORD gleWait = GetLastError();
++
++ if (gleWait == ERROR_SEM_TIMEOUT)
return IPC_STATE__NOT_LISTENING;
-+ gle = GetLastError();
+ trace2_data_intmax("ipc-debug", NULL,
+ "connect/waitpipe/gle",
-+ (intmax_t)gle);
++ (intmax_t)gleWait);
+
return IPC_STATE__OTHER_ERROR;
}
5: 5eadf719295 = 5: faf6034848e simple-ipc/ipc-win32: add Windows ACL to named pipe
6: f97038a563d ! 6: 0822118c4b5 run-command: create start_bg_command
@@ run-command.c: void prepare_other_repo_env(struct strvec *env_array, const char
+ time_t time_limit;
+
+ /*
-+ * Silently disallow child cleanup -- even if requested.
-+ * The child process should persist in the background
-+ * and possibly/probably after this process exits. That
-+ * is, don't kill the child during our atexit routine.
++ * We do not allow clean-on-exit because the child process
++ * should persist in the background and possibly/probably
++ * after this process exits. So we don't want to kill the
++ * child during our atexit routine.
+ */
-+ cmd->clean_on_exit = 0;
++ if (cmd->clean_on_exit)
++ BUG("start_bg_command() does not allow non-zero clean_on_exit");
++
++ if (!cmd->trace2_child_class)
++ cmd->trace2_child_class = "background";
+
+ ret = start_command(cmd);
+ if (ret) {
@@ run-command.c: void prepare_other_repo_env(struct strvec *env_array, const char
+wait:
+ pid_seen = waitpid(cmd->pid, &wait_status, WNOHANG);
+
-+ if (pid_seen == 0) {
++ if (!pid_seen) {
+ /*
+ * The child is currently running. Ask the callback
+ * if the child is ready to do work or whether we
+ * should keep waiting for it to boot up.
+ */
-+ ret = (*wait_cb)(cb_data, cmd);
++ ret = (*wait_cb)(cmd, cb_data);
+ if (!ret) {
+ /*
+ * The child is running and "ready".
-+ *
-+ * NEEDSWORK: As we prepare to orphan (release to
-+ * the background) this child, it is not appropriate
-+ * to emit a `trace2_child_exit()` event. Should we
-+ * create a new event for this case?
+ */
++ trace2_child_ready(cmd, "ready");
+ sbgr = SBGR_READY;
+ goto done;
+ } else if (ret > 0) {
++ /*
++ * The callback said to give it more time to boot up
++ * (subject to our timeout limit).
++ */
+ time_t now;
+
+ time(&now);
@@ run-command.c: void prepare_other_repo_env(struct strvec *env_array, const char
+ * Our timeout has expired. We don't try to
+ * kill the child, but rather let it continue
+ * (hopefully) trying to startup.
-+ *
-+ * NEEDSWORK: Like the "ready" case, should we
-+ * log a custom child-something Trace2 event here?
+ */
++ trace2_child_ready(cmd, "timeout");
+ sbgr = SBGR_TIMEOUT;
+ goto done;
+ } else {
+ /*
-+ * The cb gave up on this child.
-+ *
-+ * NEEDSWORK: Like above, should we log a custom
-+ * Trace2 child-something event here?
++ * The cb gave up on this child. It is still running,
++ * but our cb got an error trying to probe it.
+ */
++ trace2_child_ready(cmd, "error");
+ sbgr = SBGR_CB_ERROR;
+ goto done;
+ }
+ }
+
-+ if (pid_seen == cmd->pid) {
++ else if (pid_seen == cmd->pid) {
+ int child_code = -1;
+
+ /*
@@ run-command.c: void prepare_other_repo_env(struct strvec *env_array, const char
+ * before becoming "ready".
+ *
+ * We try to match the behavior of `wait_or_whine()`
++ * WRT the handling of WIFSIGNALED() and WIFEXITED()
+ * and convert the child's status to a return code for
+ * tracing purposes and emit the `trace2_child_exit()`
+ * event.
++ *
++ * We do not want the wait_or_whine() error message
++ * because we will be called by client-side library
++ * routines.
+ */
+ if (WIFEXITED(wait_status))
+ child_code = WEXITSTATUS(wait_status);
@@ run-command.c: void prepare_other_repo_env(struct strvec *env_array, const char
+ goto done;
+ }
+
-+ if (pid_seen < 0 && errno == EINTR)
++ else if (pid_seen < 0 && errno == EINTR)
+ goto wait;
+
+ trace2_child_exit(cmd, -1);
@@ run-command.h: int run_processes_parallel_tr2(int n, get_next_task_fn, start_fai
void prepare_other_repo_env(struct strvec *env_array, const char *new_git_dir);
+/**
-+ * Possible return values for `start_bg_command()`.
++ * Possible return values for start_bg_command().
+ */
+enum start_bg_result {
+ /* child process is "ready" */
@@ run-command.h: int run_processes_parallel_tr2(int n, get_next_task_fn, start_fai
+};
+
+/**
-+ * Callback used by `start_bg_command()` to ask whether the
-+ * child process is ready or needs more time to become ready.
++ * Callback used by start_bg_command() to ask whether the
++ * child process is ready or needs more time to become "ready".
++ *
++ * The callback will receive the cmd and cb_data arguments given to
++ * start_bg_command().
+ *
+ * Returns 1 is child needs more time (subject to the requested timeout).
-+ * Returns 0 if child is ready.
-+ * Returns -1 on any error and cause `start_bg_command()` to also error out.
++ * Returns 0 if child is "ready".
++ * Returns -1 on any error and cause start_bg_command() to also error out.
+ */
-+typedef int(start_bg_wait_cb)(void *cb_data,
-+ const struct child_process *cmd);
++typedef int(start_bg_wait_cb)(const struct child_process *cmd, void *cb_data);
+
+/**
-+ * Start a command in the background. Wait long enough for the child to
-+ * become "ready". Capture immediate errors (like failure to start) and
-+ * any immediate exit status (such as a shutdown/signal before the child
-+ * became "ready").
++ * Start a command in the background. Wait long enough for the child
++ * to become "ready" (as defined by the provided callback). Capture
++ * immediate errors (like failure to start) and any immediate exit
++ * status (such as a shutdown/signal before the child became "ready")
++ * and return this like start_command().
++ *
++ * We run a custom wait loop using the provided callback to wait for
++ * the child to start and become "ready". This is limited by the given
++ * timeout value.
++ *
++ * If the child does successfully start and become "ready", we orphan
++ * it into the background.
+ *
-+ * This is a combination of `start_command()` and `finish_command()`, but
-+ * with a custom `wait_or_whine()` that allows the caller to define when
-+ * the child is "ready".
++ * The caller must not call finish_command().
+ *
-+ * The caller does not need to call `finish_command()`.
++ * The opaque cb_data argument will be forwarded to the callback for
++ * any instance data that it might require. This may be NULL.
+ */
+enum start_bg_result start_bg_command(struct child_process *cmd,
+ start_bg_wait_cb *wait_cb,
7: 57f29feaadb ! 7: 6b7a058284b t/helper/simple-ipc: convert test-simple-ipc to use start_bg_command
@@ Commit message
Convert test helper to use `start_bg_command()` when spawning a server
daemon in the background rather than blocks of platform-specific code.
+ Also, while here, remove _() translation around error messages since
+ this is a test helper and not Git code.
+
Signed-off-by: Jeff Hostetler [off-list ref]
## t/helper/test-simple-ipc.c ##
@@ t/helper/test-simple-ipc.c
#ifndef SUPPORTS_SIMPLE_IPC
int cmd__simple_ipc(int argc, const char **argv)
@@ t/helper/test-simple-ipc.c: static int daemon__run_server(void)
+ */
+ ret = ipc_server_run(cl_args.path, &opts, test_app_cb, (void*)&my_app_data);
+ if (ret == -2)
+- error(_("socket/pipe already in use: '%s'"), cl_args.path);
++ error("socket/pipe already in use: '%s'", cl_args.path);
+ else if (ret == -1)
+- error_errno(_("could not start server on: '%s'"), cl_args.path);
++ error_errno("could not start server on: '%s'", cl_args.path);
+
return ret;
}
@@ t/helper/test-simple-ipc.c: static int daemon__run_server(void)
- close(1);
- close(2);
- sanitize_stdfds();
-+static int bg_wait_cb(void *cb_data, const struct child_process *cp)
++static int bg_wait_cb(const struct child_process *cp, void *cb_data)
+{
+ int s = ipc_get_active_state(cl_args.path);
@@ t/helper/test-simple-ipc.c: static int daemon__run_server(void)
/*
* This process will run a quick probe to see if a simple-ipc server
* is active on this path.
+@@ t/helper/test-simple-ipc.c: static int client__stop_server(void)
+
+ time(&now);
+ if (now > time_limit)
+- return error(_("daemon has not shutdown yet"));
++ return error("daemon has not shutdown yet");
+ }
+ }
+
--
gitgitgadget
From: Jeff Hostetler via GitGitGadget <hidden> Date: 2021-09-20 15:36:26
From: Jeff Hostetler <redacted>
Create "child_ready" event to capture the state of a child process
created in the background.
When a child command is started a "child_start" event is generated in
the Trace2 log. For normal synchronous children, a "child_exit" event
is later generated when the child exits or is terminated. The two events
include information, such as the "child_id" and "pid", to allow post
analysis to match-up the command line and exit status.
When a child is started in the background (and may outlive the parent
process), it is not possible for the parent to emit a "child_exit"
event. Create a new "child_ready" event to indicate whether the
child was successfully started. Also include the "child_id" and "pid"
to allow similar post processing.
This will be used in a later commit with the new "start_bg_command()".
Signed-off-by: Jeff Hostetler <redacted>
---
Documentation/technical/api-trace2.txt | 40 ++++++++++++++++++++++++++
trace2.c | 31 ++++++++++++++++++++
trace2.h | 25 ++++++++++++++++
trace2/tr2_tgt.h | 5 ++++
trace2/tr2_tgt_event.c | 22 ++++++++++++++
trace2/tr2_tgt_normal.c | 14 +++++++++
trace2/tr2_tgt_perf.c | 15 ++++++++++
7 files changed, 152 insertions(+)
@@ -613,6 +613,46 @@ stopping after the waitpid() and includes OS process creation overhead). So this time will be slightly larger than the atexit time reported by the child process itself.+`"child_ready"`::+ This event is generated after the current process has started+ a background process and released all handles to it.+++------------+{+ "event":"child_ready",+ ...+ "child_id":2,+ "pid":14708, # child PID+ "ready":"ready", # child ready state+ "t_rel":0.110605 # observed run-time of child process+}+------------+++Note that the session-id of the child process is not available to+the current/spawning process, so the child's PID is reported here as+a hint for post-processing. (But it is only a hint because the child+process may be a shell script which doesn't have a session-id.)+++This event is generated after the child is started in the background+and given a little time to boot up and start working. If the child+startups normally and while the parent is still waiting, the "ready"+field will have the value "ready".+If the child is too slow to start and the parent times out, the field+will have the value "timeout".+If the child starts but the parent is unable to probe it, the field+will have the value "error".+++After the parent process emits this event, it will release all of its+handles to the child process and treat the child as a background+daemon. So even if the child does eventually finish booting up,+the parent will not emit an updated event.+++Note that the `t_rel` field contains the observed run time in seconds+when the parent released the child process into the background.+The child is assumed to be a long-running daemon process and may+outlive the parent process. So the parent's child event times should+not be compared to the child's atexit times.+ `"exec"`:: This event is generated before git attempts to `exec()` another command rather than starting a child process.
From: Jeff Hostetler via GitGitGadget <hidden> Date: 2021-09-20 15:36:28
From: Jeff Hostetler <redacted>
Add `command_len` argument to the Simple IPC API.
In my original Simple IPC API, I assumed that the request would always
be a null-terminated string of text characters. The `command`
argument was just a `const char *`.
I found a caller that would like to pass a binary command to the
daemon, so I am amending the Simple IPC API to receive `const char
*command, size_t command_len` arguments.
I considered changing the `command` argument to be a `void *`, but the
IPC layer simply passes it to the pkt-line layer which takes a `const
char *`, so to avoid confusion I left it as is.
Note, the response side has always been a `struct strbuf` which
includes the buffer and length, so we already support returning a
binary answer. (Yes, it feels a little weird returning a binary
buffer in a `strbuf`, but it works.)
Signed-off-by: Jeff Hostetler <redacted>
---
compat/simple-ipc/ipc-unix-socket.c | 14 +++++++-----
compat/simple-ipc/ipc-win32.c | 14 +++++++-----
simple-ipc.h | 7 ++++--
t/helper/test-simple-ipc.c | 34 +++++++++++++++++++----------
4 files changed, 46 insertions(+), 23 deletions(-)
@@ -176,7 +177,7 @@ int ipc_client_send_command_to_connection(trace2_region_enter("ipc-client","send-command",NULL);-if(write_packetized_from_buf_no_flush(message,strlen(message),+if(write_packetized_from_buf_no_flush(message,message_len,connection->fd)<0||packet_flush_gently(connection->fd)<0){ret=error(_("could not send IPC command"));
@@ -216,7 +217,7 @@ int ipc_client_send_command_to_connection(trace2_region_enter("ipc-client","send-command",NULL);-if(write_packetized_from_buf_no_flush(message,strlen(message),+if(write_packetized_from_buf_no_flush(message,message_len,connection->fd)<0||packet_flush_gently(connection->fd)<0){ret=error(_("could not send IPC command"));
@@ -112,7 +112,7 @@ static int app__slow_command(ipc_server_reply_cb *reply_cb,/**Theclientsentacommandfollowedbya(possiblyvery)largebuffer.*/-staticintapp__sendbytes_command(constchar*received,+staticintapp__sendbytes_command(constchar*received,size_treceived_len,ipc_server_reply_cb*reply_cb,structipc_server_reply_data*reply_data){
@@ -123,6 +123,13 @@ static int app__sendbytes_command(const char *received,interrs=0;intret;+/*+*Thetestissetuptosend:+*"sendbytes"SP<n*char>+*/+if(received_len<strlen("sendbytes "))+BUG("received_len is short in app__sendbytes_command");+if(skip_prefix(received,"sendbytes ",&p))len_ballast=strlen(p);
From: Jeff Hostetler via GitGitGadget <hidden> Date: 2021-09-20 15:36:29
From: Jeff Hostetler <redacted>
From: Carlo Marcelo Arenas Belón <redacted>
Move the declartion of the `enum ipc_active_state` type outside of
the SUPPORTS_SIMPLE_IPC ifdef.
A later commit will introduce the `fsmonitor_ipc__*()` API and stub in
a "mock" implementation that requires this enum in some function
signatures.
Signed-off-by: Carlo Marcelo Arenas Belón <redacted>
Signed-off-by: Jeff Hostetler <redacted>
---
simple-ipc.h | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
From: Jeff Hostetler via GitGitGadget <hidden> Date: 2021-09-20 15:36:33
From: Jeff Hostetler <redacted>
Create a variation of `run_command()` and `start_command()` to launch a command
into the background and optionally wait for it to become "ready" before returning.
Signed-off-by: Jeff Hostetler <redacted>
---
run-command.c | 129 ++++++++++++++++++++++++++++++++++++++++++++++++++
run-command.h | 57 ++++++++++++++++++++++
2 files changed, 186 insertions(+)
@@ -496,4 +496,61 @@ int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,*/voidprepare_other_repo_env(structstrvec*env_array,constchar*new_git_dir);+/**+*Possiblereturnvaluesforstart_bg_command().+*/+enumstart_bg_result{+/* child process is "ready" */+SBGR_READY=0,++/* child process could not be started */+SBGR_ERROR,++/* callback error when testing for "ready" */+SBGR_CB_ERROR,++/* timeout expired waiting for child to become "ready" */+SBGR_TIMEOUT,++/* child process exited or was signalled before becomming "ready" */+SBGR_DIED,+};++/**+*Callbackusedbystart_bg_command()toaskwhetherthe+*childprocessisreadyorneedsmoretimetobecome"ready".+*+*Thecallbackwillreceivethecmdandcb_dataargumentsgivento+*start_bg_command().+*+*Returns1ischildneedsmoretime(subjecttotherequestedtimeout).+*Returns0ifchildis"ready".+*Returns-1onanyerrorandcausestart_bg_command()toalsoerrorout.+*/+typedefint(start_bg_wait_cb)(conststructchild_process*cmd,void*cb_data);++/**+*Startacommandinthebackground.Waitlongenoughforthechild+*tobecome"ready"(asdefinedbytheprovidedcallback).Capture+*immediateerrors(likefailuretostart)andanyimmediateexit+*status(suchasashutdown/signalbeforethechildbecame"ready")+*andreturnthislikestart_command().+*+*Werunacustomwaitloopusingtheprovidedcallbacktowaitfor+*thechildtostartandbecome"ready".Thisislimitedbythegiven+*timeoutvalue.+*+*Ifthechilddoessuccessfullystartandbecome"ready",weorphan+*itintothebackground.+*+*Thecallermustnotcallfinish_command().+*+*Theopaquecb_dataargumentwillbeforwardedtothecallbackfor+*anyinstancedatathatitmightrequire.ThismaybeNULL.+*/+enumstart_bg_resultstart_bg_command(structchild_process*cmd,+start_bg_wait_cb*wait_cb,+void*cb_data,+unsignedinttimeout_sec);+#endif
From: Jeff Hostetler via GitGitGadget <hidden> Date: 2021-09-20 15:36:35
From: Jeff Hostetler <redacted>
Set an ACL on the named pipe to allow the well-known group EVERYONE
to read and write to the IPC server's named pipe. In the event that
the daemon was started with elevation, allow non-elevated clients
to communicate with the daemon.
Signed-off-by: Jeff Hostetler <redacted>
---
compat/simple-ipc/ipc-win32.c | 140 +++++++++++++++++++++++++++++++---
1 file changed, 129 insertions(+), 11 deletions(-)
From: Jeff Hostetler via GitGitGadget <hidden> Date: 2021-09-20 15:36:37
From: Jeff Hostetler <redacted>
Convert test helper to use `start_bg_command()` when spawning a server
daemon in the background rather than blocks of platform-specific code.
Also, while here, remove _() translation around error messages since
this is a test helper and not Git code.
Signed-off-by: Jeff Hostetler <redacted>
---
t/helper/test-simple-ipc.c | 199 ++++++++-----------------------------
1 file changed, 43 insertions(+), 156 deletions(-)
@@ -267,185 +268,71 @@ static int daemon__run_server(void)*/ret=ipc_server_run(cl_args.path,&opts,test_app_cb,(void*)&my_app_data);if(ret==-2)-error(_("socket/pipe already in use: '%s'"),cl_args.path);+error("socket/pipe already in use: '%s'",cl_args.path);elseif(ret==-1)-error_errno(_("could not start server on: '%s'"),cl_args.path);+error_errno("could not start server on: '%s'",cl_args.path);returnret;}-#ifndef GIT_WINDOWS_NATIVE-/*-*Thisisadaptedfrom`daemonize()`.Use`fork()`todirectlycreateand-*runthedaemoninachildprocess.-*/-staticintspawn_server(pid_t*pid)-{-structipc_server_optsopts={-.nr_threads=cl_args.nr_threads,-};+staticstart_bg_wait_cbbg_wait_cb;-*pid=fork();--switch(*pid){-case0:-if(setsid()==-1)-error_errno(_("setsid failed"));-close(0);-close(1);-close(2);-sanitize_stdfds();+staticintbg_wait_cb(conststructchild_process*cp,void*cb_data)+{+ints=ipc_get_active_state(cl_args.path);-returnipc_server_run(cl_args.path,&opts,test_app_cb,-(void*)&my_app_data);+switch(s){+caseIPC_STATE__LISTENING:+/* child is "ready" */+return0;-case-1:-returnerror_errno(_("could not spawn daemon in the background"));+caseIPC_STATE__NOT_LISTENING:+caseIPC_STATE__PATH_NOT_FOUND:+/* give child more time */+return1;default:-return0;+caseIPC_STATE__INVALID_PATH:+caseIPC_STATE__OTHER_ERROR:+/* all the time in world won't help */+return-1;}}-#else-/*-*Conceptuallylike`daemonize()`butdifferentbecauseWindowsdoesnot-*have`fork(2)`.SpawnanormalWindowschildprocessbutwithoutthe-*limitationsof`start_command()`and`finish_command()`.-*/-staticintspawn_server(pid_t*pid)-{-chartest_tool_exe[MAX_PATH];-structstrvecargs=STRVEC_INIT;-intin,out;--GetModuleFileNameA(NULL,test_tool_exe,MAX_PATH);--in=open("/dev/null",O_RDONLY);-out=open("/dev/null",O_WRONLY);--strvec_push(&args,test_tool_exe);-strvec_push(&args,"simple-ipc");-strvec_push(&args,"run-daemon");-strvec_pushf(&args,"--name=%s",cl_args.path);-strvec_pushf(&args,"--threads=%d",cl_args.nr_threads);--*pid=mingw_spawnvpe(args.v[0],args.v,NULL,NULL,in,out,out);-close(in);-close(out);--strvec_clear(&args);-if(*pid<0)-returnerror(_("could not spawn daemon in the background"));--return0;-}-#endif--/*-*Thisisadaptedfrom`wait_or_whine()`.Watchthechildprocessand-*letitgetstartedandbeginlisteningforrequestsonthesocket-*beforereportingoursuccess.-*/-staticintwait_for_server_startup(pid_tpid_child)+staticintdaemon__start_server(void){-intstatus;-pid_tpid_seen;-enumipc_active_states;-time_ttime_limit,now;+structchild_processcp=CHILD_PROCESS_INIT;+enumstart_bg_resultsbgr;-time(&time_limit);-time_limit+=cl_args.max_wait_sec;+strvec_push(&cp.args,"test-tool");+strvec_push(&cp.args,"simple-ipc");+strvec_push(&cp.args,"run-daemon");+strvec_pushf(&cp.args,"--name=%s",cl_args.path);+strvec_pushf(&cp.args,"--threads=%d",cl_args.nr_threads);-for(;;){-pid_seen=waitpid(pid_child,&status,WNOHANG);+cp.no_stdin=1;+cp.no_stdout=1;+cp.no_stderr=1;-if(pid_seen==-1)-returnerror_errno(_("waitpid failed"));+sbgr=start_bg_command(&cp,bg_wait_cb,NULL,cl_args.max_wait_sec);-elseif(pid_seen==0){-/*-*Thechildisstillrunning(thisshouldbe-*thenormalcase).Trytoconnecttoiton-*thesocketandseeifitisreadyfor-*business.-*-*Ifthereisanotherdaemonalreadyrunning,-*ourchildwillfailtostart(possibly-*afteratimeoutonthelock),butwedon't-*care(whoresponds)ifthesocketislive.-*/-s=ipc_get_active_state(cl_args.path);-if(s==IPC_STATE__LISTENING)-return0;--time(&now);-if(now>time_limit)-returnerror(_("daemon not online yet"));--continue;-}+switch(sbgr){+caseSBGR_READY:+return0;-elseif(pid_seen==pid_child){-/*-*Thenewchilddaemonprocessshutdownwhile-*itwasstartingup,soitisnotlistening-*onthesocket.-*-*Trytopingthesocketintheoddchance-*thatanotherdaemonstarted(orwasalready-*running)whileourchildwasstarting.-*-*Again,wedon'tcarewhoservicesthesocket.-*/-s=ipc_get_active_state(cl_args.path);-if(s==IPC_STATE__LISTENING)-return0;+default:+caseSBGR_ERROR:+caseSBGR_CB_ERROR:+returnerror("daemon failed to start");-/*-*Wedon'tcareabouttheWEXITSTATUS()nor-*anyoftheWIF*(status)valuesbecause-*`cmd__simple_ipc()`doesthe`!!result`-*trickonallfunctionreturnvalues.-*-*Soitissufficienttojustreportthe-*earlyshutdownasanerror.-*/-returnerror(_("daemon failed to start"));-}+caseSBGR_TIMEOUT:+returnerror("daemon not online yet");-else-returnerror(_("waitpid is confused"));+caseSBGR_DIED:+returnerror("daemon terminated");}}-/*-*Thisprocesswillstartasimple-ipcserverinabackgroundprocessand-*waitforittobecomeready.Thisislike`daemonize()`butgivesus-*morecontrolandbettererrorreporting(andmakesiteasiertowrite-*unittests).-*/-staticintdaemon__start_server(void)-{-pid_tpid_child;-intret;--/*-*Runtheactualdaemoninabackgroundprocess.-*/-ret=spawn_server(&pid_child);-if(pid_child<=0)-returnret;--/*-*Lettheparentwaitforthechildprocesstogetstarted-*andbeginlisteningforrequestsonthesocket.-*/-ret=wait_for_server_startup(pid_child);--returnret;-}-/**Thisprocesswillrunaquickprobetoseeifasimple-ipcserver*isactiveonthispath.
@@ -548,7 +435,7 @@ static int client__stop_server(void)time(&now);if(now>time_limit)-returnerror(_("daemon has not shutdown yet"));+returnerror("daemon has not shutdown yet");}}
On Mon, Sep 20 2021, Jeff Hostetler via GitGitGadget wrote:
Here is V2 of Part 1 of my Builtin FSMonitor series.
Changes since V1 include:
* Drop the Trace2 memory leak.
* Added a new "child_ready" event to Trace2 as an alternative to the
"child_exit" event for background processes.
* Convert the Trace2-related NEEDSWORK items in "start_bg_command()" to use
the new "child_ready" event.
* Various minor code and documentation cleanups.
I see 7/7 still has a pattern you included only to make a compiler error
better. I noted in
https://lore.kernel.org/git/87ilyycko3.fsf@evledraar.gmail.com/ that it
make the error worse, on at least clang. You didn't note which compiler
you were massaging, presumably MSVC.
I think that's a relatively small matter, but it *is* one I know about,
and there's no reply there, mention of it being unaddressed here or in
the commit message.
I haven't gone back & re-read v1 and seen if there's more unaddresed
feedback from others, instead I wanted to encourage you to provide such
a summary.
It really helps when a series is re-rolled to aid review both for
newcomers and returning reviewers. I really don't care about "getting my
way" on such a minor thing.
But it is frustrating to have the state of a re-roll be observably
indistinguishable from one's E-Mail not having been received, when
that's the case you've got to go back and re-read the thread, scour the
range-diff, and generalyl do a lot of work that the person doing the
re-roll has done, but either didn't keep notes, or didn't share them.
Personally I'm in the habit of "flagging" (starring in GMail terms)
E-Mails with outstanding unaddresed comments I get on my own topics,
then when I re-roll them I look at the thread, and "unwind the stack" as
it were by removing flags on E-Mails that I've either addressed via
updated commit messages, or added a note to a WIP cover letter.
E.g. here (just an example that includes Taylor, since he reviewed v1
here) is a case where Taylor suggested something that I didn't go for,
but i'd like to think noting it helped him catch up:
https://lore.kernel.org/git/cover-v4-0.5-00000000000-20210921T131003Z-avarab@gmail.com/
All the best, just trying to make the reviewer & re-rolling process
better for everyone.
On Mon, Sep 20 2021, Jeff Hostetler via GitGitGadget wrote:
+ switch (sbgr) {
+ case SBGR_READY:
+ return 0;
- else if (pid_seen == pid_child) {
- /*
- * The new child daemon process shutdown while
- * it was starting up, so it is not listening
- * on the socket.
- *
- * Try to ping the socket in the odd chance
- * that another daemon started (or was already
- * running) while our child was starting.
- *
- * Again, we don't care who services the socket.
- */
- s = ipc_get_active_state(cl_args.path);
- if (s == IPC_STATE__LISTENING)
- return 0;
+ default:
+ case SBGR_ERROR:
+ case SBGR_CB_ERROR:
+ return error("daemon failed to start");
There was a discussion on v1 about the "default" being redundant here
and hiding future compiler checks, this is another "not sure what you
thought of that" case (per [1]).
Interestingly in this case if I drop the "default" my local gcc
uncharacteristically complains about a missing "return" in this
function, but clang doesn't. I needed to add a BUG() to shut up the
former. Maybe I'm wrong, but perhaps it's a sign of some deeper
trouble. This is with gcc/clang 10.2.1-6/11.0.1-2.
1. https://lore.kernel.org/git/87v92r49mt.fsf@evledraar.gmail.com/
I played with the diff below on top of this, I can't remember if it was
noted already, but the way you declare function ptrs and use them isn't
the usual style:
-- >8 --
@@ -275,9 +275,7 @@ static int daemon__run_server(void)returnret;}-staticstart_bg_wait_cbbg_wait_cb;--staticintbg_wait_cb(conststructchild_process*cp,void*cb_data)+staticintbg_wait_cb(conststructchild_process*cp,void*cb_data,intfoo){ints=ipc_get_active_state(cl_args.path);
@@ -319,9 +317,8 @@ static int daemon__start_server(void)switch(sbgr){caseSBGR_READY:return0;--default:caseSBGR_ERROR:+return0;caseSBGR_CB_ERROR:returnerror("daemon failed to start");
@@ -331,6 +328,7 @@ static int daemon__start_server(void)caseSBGR_DIED:returnerror("daemon terminated");}+BUG("unreachable");}/*
From: Jeff Hostetler <hidden> Date: 2021-09-23 17:13:00
On 9/23/21 10:33 AM, Ævar Arnfjörð Bjarmason wrote:
On Mon, Sep 20 2021, Jeff Hostetler via GitGitGadget wrote:
quoted
Here is V2 of Part 1 of my Builtin FSMonitor series.
Changes since V1 include:
* Drop the Trace2 memory leak.
* Added a new "child_ready" event to Trace2 as an alternative to the
"child_exit" event for background processes.
* Convert the Trace2-related NEEDSWORK items in "start_bg_command()" to use
the new "child_ready" event.
* Various minor code and documentation cleanups.
I see 7/7 still has a pattern you included only to make a compiler error
better. I noted in
https://lore.kernel.org/git/87ilyycko3.fsf@evledraar.gmail.com/ that it
make the error worse, on at least clang. You didn't note which compiler
you were massaging, presumably MSVC.
I've been holding my tongue for days on this issue and hoping a third
party would step in an render an opinion one way or another.
Too me, a forward declaration seemed like no big deal and it does
have value as I tried to explain. And frankly, it felt a little bit
like bike-shedding and was trying to avoid that again.
The error message I quoted was from Clang v11.0.3. My forward
declaration of the function prior to the actual definition of
the function causes the compiler to stop at the function definition
and complain with a short message saying that the function itself
is incorrectly defined and doesn't match the typedef that it is
associated with.
When I use MSVC I get a similar error at the function definition.
When I use GCC I get error messages at both the function definition
and the usage in the call.
Additionally, the forward declaration states that the function is
associated with that typedef (something that is otherwise implicit
and may be hard to discover (more on that in a minute)).
And it doesn't require a reference to the function pointer (either
on the right side of an assignment, a vtable initialization, or passing
it in a function call) to flag the error. We always get the error
at the point of the definition.
The error message in your example is, I feel, worse than mine.
It splats 2 different function signatures -- only one of which has
the typedef name -- in a large, poorly wrapped brick of text.
Yes, your error message does print corresponding arg in the function
prototype of "start_bg_command()" that doesn't agree with the symbol
used at the call-site, but that is much later than where the actual
error occurred. And if the forward declaration were present, you'd
already know that back up at the definition, right.
Let's look at this from another point of view.
Suppose for example we have two function prototypes with the same
signature. Perhaps they describe groups of functions with different
semantics -- the fact that they have the same argument list and return
type is just a coincidence.
typedef int(t_fn_1)(int);
typedef int(t_fn_2)(int);
And then declare one or more instances of functions in those groups:
int foo_a(int x) { ... }
int foo_b(int x) { ... }
int foo_c(int x) { ... }
int foo_d(int x) { ... }
int foo_e(int x) { ... }
int foo_f(int x) { ... }
int foo_g(int x) { ... }
Which of those functions should be associated with "t_fn_1" and which
with "t_fn_2"? Again, they all have the same signature, but different
semantics. The author knows when they wrote the code, but it may be
hard to automatically determine later.
If I then have a function like start_bg_command() that receives a
function pointer:
int test(..., t_fn_1 *fn, ...) { ... }
In C -- even with the use of forward function declarations -- the
compiler won't complain if you pass test() a pointer of type t_fn_2
-- as long a t_fn_1 and t_fn_2 have the same signature.
But it does give the human a chance to catch the error. Of if we
later change the function signature in the t_fn_1 typedef, we will
automatically get a list of which of those foo_x functions do and
do not need to be updated.
Anyway, I've soapboxed on this enough. I think it is a worthy
feature for the price.
Jeff
It did, but to be honest I would have been totally fine with you
mentioning the changes you did incorporate into the rerolled version,
and omitting mention of any insignificant suggestions you decided to
ignore.
In other words, when I got to the same spot in the rerolled version, I
would have either thought "looks like Ævar didn't take my suggestion,
OK" or not have remembered it in the first place. Either way, the point
was trivial enough that I didn't bother to pursue it further in that
thread.
And I think that's what is happening here, too. Yes, I find the forward
declaration useless on GCC, though it appears to be helpful on MSVC and
hurtful on clang. Even if it does produce a strictly worse error message
on clang, do we really care? It may cause some mild inconvenience for a
developer later on, but I find it highly unlikely that it would allow us
to ship a bug that wouldn't have been caught one way or another during
development.
So I was a little disappointed to see such a back-and-forth about this
quite trivial point. I realize that I'm piling on here by adding my
two-cents, but I think it's worth it to ask ourselves more often what
points we're willing to concede and which are worth advocating more
strongly for.
Thanks,
Taylor
From: Jeff Hostetler <hidden> Date: 2021-09-23 17:58:55
On 9/23/21 11:03 AM, Ævar Arnfjörð Bjarmason wrote:
On Mon, Sep 20 2021, Jeff Hostetler via GitGitGadget wrote:
quoted
+ switch (sbgr) {
+ case SBGR_READY:
+ return 0;
- else if (pid_seen == pid_child) {
- /*
- * The new child daemon process shutdown while
- * it was starting up, so it is not listening
- * on the socket.
- *
- * Try to ping the socket in the odd chance
- * that another daemon started (or was already
- * running) while our child was starting.
- *
- * Again, we don't care who services the socket.
- */
- s = ipc_get_active_state(cl_args.path);
- if (s == IPC_STATE__LISTENING)
- return 0;
+ default:
+ case SBGR_ERROR:
+ case SBGR_CB_ERROR:
+ return error("daemon failed to start");
There was a discussion on v1 about the "default" being redundant here
and hiding future compiler checks, this is another "not sure what you
thought of that" case (per [1]).
Interestingly in this case if I drop the "default" my local gcc
uncharacteristically complains about a missing "return" in this
function, but clang doesn't. I needed to add a BUG() to shut up the
former. Maybe I'm wrong, but perhaps it's a sign of some deeper
trouble. This is with gcc/clang 10.2.1-6/11.0.1-2.
The issue of whether C needs a "default" case in switch statements
on an enum is one I didn't have bandwidth to think about (and is
completely independent of my series).
I was thinking that as a later task, someone could investigate which
compilers do and do not generate errors for missing enum values in
the switch. Perhaps that leads to a macro in config.mak.uname on
a system-by-system basis that "does the right thing".
Then one could have something like:
switch (e) {
DEFAULT_HANDLER;
case e1: ...
case e2: ...
}
By defining the typedef WITHOUT the star, we get a function type.
We can then use it for forward function declarations.
But additionally, when declare a function that takes a function
pointer or when we define a vtable of function pointers, they look
like pointers.
start_bg_wait_cb *pfn = my_cb;
int foo(struct child_process *cmd, start_bg_wait_cb *cb);
Or if we look a the target vtable in Trace2. This looks like
a structure of (function) pointers.
struct tr2_tgt {
tr2_tgt_init_t *pfn_init;
tr2_tgt_term_t *pfn_term;
...
};
So I prefer to leave the star out of function typedef and then
we can use the typedef in both contexts.
Jeff
On 9/23/21 10:33 AM, Ævar Arnfjörð Bjarmason wrote:
quoted
On Mon, Sep 20 2021, Jeff Hostetler via GitGitGadget wrote:
quoted
Here is V2 of Part 1 of my Builtin FSMonitor series.
Changes since V1 include:
* Drop the Trace2 memory leak.
* Added a new "child_ready" event to Trace2 as an alternative to the
"child_exit" event for background processes.
* Convert the Trace2-related NEEDSWORK items in "start_bg_command()" to use
the new "child_ready" event.
* Various minor code and documentation cleanups.
I see 7/7 still has a pattern you included only to make a compiler
error
better. I noted in
https://lore.kernel.org/git/87ilyycko3.fsf@evledraar.gmail.com/ that it
make the error worse, on at least clang. You didn't note which compiler
you were massaging, presumably MSVC.
I've been holding my tongue for days on this issue and hoping a third
party would step in an render an opinion one way or another.
Too me, a forward declaration seemed like no big deal and it does
have value as I tried to explain. And frankly, it felt a little bit
like bike-shedding and was trying to avoid that again.
I agree with you that it's no big deal in the end,
I thought I made it clear in [off-list ref] but
the main thing I'm commenting on is not that I or anyone else suggested
Y over X, and you said nah and went for X in the end.
That's fine, I mean, depending on the comment/issue etc. it's something
other reviewers & Junio can draw their own conclusions about.
What I am saying that it's much better for review of iterations of
patches in general, and especially of a complex multi-part series if
reviewers don't have to read the cover letter of vX and wonder what's
omitted/unaddressed in the V(X-1) comments, and then go and re-read the
discussion themselves. It's not the "nah", but that the "nah" is
implicit and only apparent when sending an E-Mail like this.
Of course that's never perfect, you can't summarize every point
etc. Personally I try to do this, but I've sometimes noticed after the
fact that I've gotten it wrong etc.
In the end I and I think anyone else offering their time to review
things is trying to move the relevant topic forward in one way or
another. I'd much rather spend my time on a vX discussing new things &
getting the thing closer to merge-able state, than re-reading all of
v(X-1) & effectively coming up with my own cover letter summary in my
head or in my own notes as I read along.
Anyway, sorry about the bikeshedding getting out of hand, and what seems
to have been at least partially a misunderstanding in the last couple of
E-Mails between us, but the above is all I was going for.
The error message I quoted was from Clang v11.0.3. My forward
declaration of the function prior to the actual definition of
the function causes the compiler to stop at the function definition
and complain with a short message saying that the function itself
is incorrectly defined and doesn't match the typedef that it is
associated with.
When I use MSVC I get a similar error at the function definition.
When I use GCC I get error messages at both the function definition
and the usage in the call.
Additionally, the forward declaration states that the function is
associated with that typedef (something that is otherwise implicit
and may be hard to discover (more on that in a minute)).
And it doesn't require a reference to the function pointer (either
on the right side of an assignment, a vtable initialization, or passing
it in a function call) to flag the error. We always get the error
at the point of the definition.
The error message in your example is, I feel, worse than mine.
It splats 2 different function signatures -- only one of which has
the typedef name -- in a large, poorly wrapped brick of text.
For what it's worth any poor wrapping is my fault, I have a relatively
wide terminal and re-wrapped this when composing the E-Mail. I think
both GCC & Clang (and most other mature compilers) would give the person
getting the error sane wrapping based on their $COLUMNS.
Yes, your error message does print corresponding arg in the function
prototype of "start_bg_command()" that doesn't agree with the symbol
used at the call-site, but that is much later than where the actual
error occurred. And if the forward declaration were present, you'd
already know that back up at the definition, right.
Let's look at this from another point of view.
Suppose for example we have two function prototypes with the same
signature. Perhaps they describe groups of functions with different
semantics -- the fact that they have the same argument list and return
type is just a coincidence.
typedef int(t_fn_1)(int);
typedef int(t_fn_2)(int);
And then declare one or more instances of functions in those groups:
int foo_a(int x) { ... }
int foo_b(int x) { ... }
int foo_c(int x) { ... }
int foo_d(int x) { ... }
int foo_e(int x) { ... }
int foo_f(int x) { ... }
int foo_g(int x) { ... }
Which of those functions should be associated with "t_fn_1" and which
with "t_fn_2"? Again, they all have the same signature, but different
semantics. The author knows when they wrote the code, but it may be
hard to automatically determine later.
If I then have a function like start_bg_command() that receives a
function pointer:
int test(..., t_fn_1 *fn, ...) { ... }
In C -- even with the use of forward function declarations -- the
compiler won't complain if you pass test() a pointer of type t_fn_2
-- as long a t_fn_1 and t_fn_2 have the same signature.
But it does give the human a chance to catch the error. Of if we
later change the function signature in the t_fn_1 typedef, we will
automatically get a list of which of those foo_x functions do and
do not need to be updated.
Anyway, I've soapboxed on this enough. I think it is a worthy
feature for the price.
Code in git.git generally just declares say an "int foo(int)" and leaves
it at passing the "foo", we're not concerned about that "foo" just so
happening to be passed to some other interface that takes the same
signature, certainly not something within a <1k line t/helper/* file.
We all have habits we've picked up from other codebases prior to working
on git.git. I'm not arguing that what you're describing is worse in some
abtract sense, but that there's a larger value in following conventions
within the codebase as Junio noted in his reply.
From: Jeff Hostetler <hidden> Date: 2021-09-27 13:37:25
On 9/23/21 4:47 PM, Ævar Arnfjörð Bjarmason wrote:
On Thu, Sep 23 2021, Jeff Hostetler wrote:
quoted
On 9/23/21 10:33 AM, Ævar Arnfjörð Bjarmason wrote:
quoted
On Mon, Sep 20 2021, Jeff Hostetler via GitGitGadget wrote:
quoted
Here is V2 of Part 1 of my Builtin FSMonitor series.
Changes since V1 include:
* Drop the Trace2 memory leak.
* Added a new "child_ready" event to Trace2 as an alternative to the
"child_exit" event for background processes.
* Convert the Trace2-related NEEDSWORK items in "start_bg_command()" to use
the new "child_ready" event.
* Various minor code and documentation cleanups.
I see 7/7 still has a pattern you included only to make a compiler
error
better. I noted in
https://lore.kernel.org/git/87ilyycko3.fsf@evledraar.gmail.com/ that it
make the error worse, on at least clang. You didn't note which compiler
you were massaging, presumably MSVC.
I've been holding my tongue for days on this issue and hoping a third
party would step in an render an opinion one way or another.
Too me, a forward declaration seemed like no big deal and it does
have value as I tried to explain. And frankly, it felt a little bit
like bike-shedding and was trying to avoid that again.
I agree with you that it's no big deal in the end,
I thought I made it clear in [off-list ref] but
the main thing I'm commenting on is not that I or anyone else suggested
Y over X, and you said nah and went for X in the end.
That's fine, I mean, depending on the comment/issue etc. it's something
other reviewers & Junio can draw their own conclusions about.
What I am saying that it's much better for review of iterations of
patches in general, and especially of a complex multi-part series if
reviewers don't have to read the cover letter of vX and wonder what's
omitted/unaddressed in the V(X-1) comments, and then go and re-read the
discussion themselves. It's not the "nah", but that the "nah" is
implicit and only apparent when sending an E-Mail like this.
Of course that's never perfect, you can't summarize every point
etc. Personally I try to do this, but I've sometimes noticed after the
fact that I've gotten it wrong etc.
In the end I and I think anyone else offering their time to review
things is trying to move the relevant topic forward in one way or
another. I'd much rather spend my time on a vX discussing new things &
getting the thing closer to merge-able state, than re-reading all of
v(X-1) & effectively coming up with my own cover letter summary in my
head or in my own notes as I read along.
Anyway, sorry about the bikeshedding getting out of hand, and what seems
to have been at least partially a misunderstanding in the last couple of
E-Mails between us, but the above is all I was going for.
Thanks. Yeah, email is a terrible communication medium and prone
to misunderstandings. It's easy to forget that at times, since we
spend so much time in it.
Drafting a v(X+1) cover letter is a bit of an art. It is easy to
err on the less-is-better side when trying to decide how much to
include to explain the new version vs not wanting to including
every little typo or nit.
I tend to drop / cross-off issues that I decide to ignore or not act
upon rather than report them. However, you're right, I should have
included a brief statement about not changing the stuff mentioned in
7/7, since there was a larger conversation around it. Sorry.
Jeff
From: Adam Dinwoodie <hidden> Date: 2021-11-04 19:46:35
On Monday 20 September 2021 at 03:36 pm +0000, Jeff Hostetler via GitGitGadget wrote:
From: Jeff Hostetler <redacted>
Convert test helper to use `start_bg_command()` when spawning a server
daemon in the background rather than blocks of platform-specific code.
Also, while here, remove _() translation around error messages since
this is a test helper and not Git code.
As part of testing the v2.34.0-rc0 release candidate on Cygwin, I've
found this commit -- 05881a6fc9 (t/helper/simple-ipc: convert
test-simple-ipc to use start_bg_command, 2021-09-20), according to my
bisecting -- is causing t0052.1 to fail on 32-bit Cygwin. Somehow this
is only affecting the 32-bit Cygwin build; the 64-bit build is working
as expected.
Specifically, the failure I'm seeing is as below:
I've had a look at the code changes, and cannot work out what might be
being handled differently in 32-bit and 64-bit Cygwin environments.
Given the Cygwin project is considering dropping support for 32-bit
Cygwin anyway, it might not be worth doing anything about this. But I
thought it worth reporting in case there's something obvious to folk
more familiar with this code.
From: Ramsay Jones <hidden> Date: 2021-11-04 20:14:14
Hi Adam,
On 04/11/2021 19:46, Adam Dinwoodie wrote:
On Monday 20 September 2021 at 03:36 pm +0000, Jeff Hostetler via GitGitGadget wrote:
quoted
From: Jeff Hostetler <redacted>
Convert test helper to use `start_bg_command()` when spawning a server
daemon in the background rather than blocks of platform-specific code.
Also, while here, remove _() translation around error messages since
this is a test helper and not Git code.
As part of testing the v2.34.0-rc0 release candidate on Cygwin, I've
found this commit -- 05881a6fc9 (t/helper/simple-ipc: convert
test-simple-ipc to use start_bg_command, 2021-09-20), according to my
bisecting -- is causing t0052.1 to fail on 32-bit Cygwin. Somehow this
is only affecting the 32-bit Cygwin build; the 64-bit build is working
as expected.
Hmmm, I am seeing exactly the same, but on 64-bit cygwin!
I haven't found time to look at this in detail yet (except for
what you have already done). Unfortunately, about an hour ago,
I did a 'make test' for the '-rc1' build, so I won't be able to
take a look for hours yet, ... :(
ATB,
Ramsay Jones
From: Jeff Hostetler <hidden> Date: 2021-11-08 14:58:03
On 11/4/21 3:46 PM, Adam Dinwoodie wrote:
On Monday 20 September 2021 at 03:36 pm +0000, Jeff Hostetler via GitGitGadget wrote:
quoted
From: Jeff Hostetler <redacted>
Convert test helper to use `start_bg_command()` when spawning a server
daemon in the background rather than blocks of platform-specific code.
Also, while here, remove _() translation around error messages since
this is a test helper and not Git code.
As part of testing the v2.34.0-rc0 release candidate on Cygwin, I've
found this commit -- 05881a6fc9 (t/helper/simple-ipc: convert
test-simple-ipc to use start_bg_command, 2021-09-20), according to my
bisecting -- is causing t0052.1 to fail on 32-bit Cygwin. Somehow this
is only affecting the 32-bit Cygwin build; the 64-bit build is working
as expected.
Specifically, the failure I'm seeing is as below:
I've had a look at the code changes, and cannot work out what might be
being handled differently in 32-bit and 64-bit Cygwin environments.
Given the Cygwin project is considering dropping support for 32-bit
Cygwin anyway, it might not be worth doing anything about this. But I
thought it worth reporting in case there's something obvious to folk
more familiar with this code.
How odd! Thanks for the report. I'll investigate.
Jeff
From: Johannes Schindelin <hidden> Date: 2021-11-08 23:59:53
Hi Adam,
On Thu, 4 Nov 2021, Adam Dinwoodie wrote:
On Monday 20 September 2021 at 03:36 pm +0000, Jeff Hostetler via GitGitGadget wrote:
quoted
From: Jeff Hostetler <redacted>
Convert test helper to use `start_bg_command()` when spawning a server
daemon in the background rather than blocks of platform-specific code.
Also, while here, remove _() translation around error messages since
this is a test helper and not Git code.
As part of testing the v2.34.0-rc0 release candidate on Cygwin, I've
found this commit -- 05881a6fc9 (t/helper/simple-ipc: convert
test-simple-ipc to use start_bg_command, 2021-09-20), according to my
bisecting -- is causing t0052.1 to fail on 32-bit Cygwin. Somehow this
is only affecting the 32-bit Cygwin build; the 64-bit build is working
as expected.
Specifically, the failure I'm seeing is as below:
I've had a look at the code changes, and cannot work out what might be
being handled differently in 32-bit and 64-bit Cygwin environments.
Given the Cygwin project is considering dropping support for 32-bit
Cygwin anyway, it might not be worth doing anything about this. But I
thought it worth reporting in case there's something obvious to folk
more familiar with this code.
I had a look at this and could reproduce... partially. I only managed to
make it fail every once in a while.
Digging deeper, it turns out that the `lstat()` call in
`ipc_get_active_state()` does not receive an `st_mode` indicating a
socket, but rather a file (in my tests, it was usually 0100644, but
sometimes even 0100755).
The reason is, of course, that Cygwin _emulates_ Unix sockets. What is in
the file system is just a special file, it is marked with the `system` bit
(which only exists on Windows), and its contents start with the tell-tale
`!<socket>`.
And as you might have guessed, there is a race going on between Cygwin
writing that file _and_ flipping that `system` bit, and Git trying to
access the Unix socket and encountering an unexpected file.
Now, why this only happens in your 32-bit setup, I have no idea.
In my tests, the following patch works around the issue. Could I ask you
to test it in your environment?
-- snip --
diff --git a/compat/simple-ipc/ipc-unix-socket.c
b/compat/simple-ipc/ipc-unix-socket.c
index 4e28857a0a..1c591b2adf 100644
*path)
}
/* also complain if a plain file is in the way */
+#ifdef __CYGWIN__
+ {
+ static const int delay[] = { 1, 10, 20, 40, -1 };
+ int i;
+
+ for (i = 0; S_ISREG(st.st_mode) && delay[i] > 0; i++) {
+ /*
+ * Cygwin might still be in the process of marking the
+ * underlying file as a system file.
+ */
+ sleep_millisec(delay[i]);
+ if (lstat(path, &st) == -1)
+ return IPC_STATE__INVALID_PATH;
+ }
+ }
+#endif
+
if ((st.st_mode & S_IFMT) != S_IFSOCK)
return IPC_STATE__INVALID_PATH;
-- snap --
FWIW it looks as if the loop might be a bit of an overkill, as I could not
get the code to need more than a single one-millisecond sleep, but it's
probably safer to just keep the delay loop in place as-is.
Ciao,
Dscho
From: Ramsay Jones <hidden> Date: 2021-11-09 18:53:55
On 08/11/2021 23:59, Johannes Schindelin wrote:
[snip]
I had a look at this and could reproduce... partially. I only managed to
make it fail every once in a while.
Digging deeper, it turns out that the `lstat()` call in
`ipc_get_active_state()` does not receive an `st_mode` indicating a
socket, but rather a file (in my tests, it was usually 0100644, but
sometimes even 0100755).
The reason is, of course, that Cygwin _emulates_ Unix sockets. What is in
the file system is just a special file, it is marked with the `system` bit
(which only exists on Windows), and its contents start with the tell-tale
`!<socket>`.
And as you might have guessed, there is a race going on between Cygwin
writing that file _and_ flipping that `system` bit, and Git trying to
access the Unix socket and encountering an unexpected file.
Now, why this only happens in your 32-bit setup, I have no idea.
In my tests, the following patch works around the issue. Could I ask you
to test it in your environment?
Just FYI, I just tried the patch below (on 64-bit cygwin) and this test
now works fine for me. (well, run 5 times by hand - not with --stress).
This is on windows 10 21H1 and cygwin:
$ uname -a
CYGWIN_NT-10.0 satellite 3.3.2(0.341/5/3) 2021-11-08 16:55 x86_64 Cygwin
$
[Yes, I updated last night!]
ATB,
Ramsay Jones
quoted hunk
-- snip --
diff --git a/compat/simple-ipc/ipc-unix-socket.c
b/compat/simple-ipc/ipc-unix-socket.c
index 4e28857a0a..1c591b2adf 100644
*path)
}
/* also complain if a plain file is in the way */
+#ifdef __CYGWIN__
+ {
+ static const int delay[] = { 1, 10, 20, 40, -1 };
+ int i;
+
+ for (i = 0; S_ISREG(st.st_mode) && delay[i] > 0; i++) {
+ /*
+ * Cygwin might still be in the process of marking the
+ * underlying file as a system file.
+ */
+ sleep_millisec(delay[i]);
+ if (lstat(path, &st) == -1)
+ return IPC_STATE__INVALID_PATH;
+ }
+ }
+#endif
+
if ((st.st_mode & S_IFMT) != S_IFSOCK)
return IPC_STATE__INVALID_PATH;
-- snap --
FWIW it looks as if the loop might be a bit of an overkill, as I could not
get the code to need more than a single one-millisecond sleep, but it's
probably safer to just keep the delay loop in place as-is.
Ciao,
Dscho
From: Johannes Schindelin <hidden> Date: 2021-11-09 23:01:48
Hi Ramsay,
On Tue, 9 Nov 2021, Ramsay Jones wrote:
On 08/11/2021 23:59, Johannes Schindelin wrote:
[snip]
quoted
I had a look at this and could reproduce... partially. I only managed to
make it fail every once in a while.
Digging deeper, it turns out that the `lstat()` call in
`ipc_get_active_state()` does not receive an `st_mode` indicating a
socket, but rather a file (in my tests, it was usually 0100644, but
sometimes even 0100755).
The reason is, of course, that Cygwin _emulates_ Unix sockets. What is in
the file system is just a special file, it is marked with the `system` bit
(which only exists on Windows), and its contents start with the tell-tale
`!<socket>`.
And as you might have guessed, there is a race going on between Cygwin
writing that file _and_ flipping that `system` bit, and Git trying to
access the Unix socket and encountering an unexpected file.
Now, why this only happens in your 32-bit setup, I have no idea.
In my tests, the following patch works around the issue. Could I ask you
to test it in your environment?
Just FYI, I just tried the patch below (on 64-bit cygwin) and this test
now works fine for me. (well, run 5 times by hand - not with --stress).
Very good!
I fear that it is a bit late in the -rc cycle to try to get this into the
official v2.34.0. Adam, since you are the maintainer of the Cygwin git
package, would you mind incorporating this patch into Cygwin's version of
Git?
This is on windows 10 21H1 and cygwin:
$ uname -a
CYGWIN_NT-10.0 satellite 3.3.2(0.341/5/3) 2021-11-08 16:55 x86_64 Cygwin
$
[Yes, I updated last night!]
Good thing, too: v3.3.2 fixes a critical bug in the pipe code. One symptom
was that you could not use Git Credential Manager Core as credential
helper (because Git thought that the helper had hung up, well before the
helper sent any information).
Ciao,
Dscho
ATB,
Ramsay Jones
quoted
-- snip --
diff --git a/compat/simple-ipc/ipc-unix-socket.c
b/compat/simple-ipc/ipc-unix-socket.c
index 4e28857a0a..1c591b2adf 100644
*path)
}
/* also complain if a plain file is in the way */
+#ifdef __CYGWIN__
+ {
+ static const int delay[] = { 1, 10, 20, 40, -1 };
+ int i;
+
+ for (i = 0; S_ISREG(st.st_mode) && delay[i] > 0; i++) {
+ /*
+ * Cygwin might still be in the process of marking the
+ * underlying file as a system file.
+ */
+ sleep_millisec(delay[i]);
+ if (lstat(path, &st) == -1)
+ return IPC_STATE__INVALID_PATH;
+ }
+ }
+#endif
+
if ((st.st_mode & S_IFMT) != S_IFSOCK)
return IPC_STATE__INVALID_PATH;
-- snap --
FWIW it looks as if the loop might be a bit of an overkill, as I could not
get the code to need more than a single one-millisecond sleep, but it's
probably safer to just keep the delay loop in place as-is.
Ciao,
Dscho