From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-08-30 21:34:54
tl;dr: This series contributes the Scalar command to the Git project. This
command provides an opinionated way to create and configure repositories
with a focus on very large repositories.
Background
==========
Years ago, Microsoft wanted to move the source code of the Windows operating
system to Git. The challenge there was to prove that Git could scale to
massive monorepos. The VFS for Git (formerly GVFS) project was born to take
up that challenge.
The final solution included a virtual filesystem (with both user-mode and
kernel components) and a customized fork of Git for Windows. This solution
contained several key concepts, such as only populating a portion of the
working directory, demand-fetching blobs, and performing periodic repo
maintenance in the background. However, the required kernel drivers made it
difficult to port the solution to other platforms.
But it was realized that many of these key concepts were independent of the
actual VFS and its projection of the working directory. The Scalar project
was created to make that separation, refine the key concepts, and then
extract those features into the new Scalar command.
The present
===========
The Scalar project provides a completely functional non-virtual experience
for monorepos. But why stop there. The Scalar project was designed to be a
self-destructing vehicle to allow those key concepts to be moved into core
Git itself for the benefit of all. For example, partial clone,
sparse-checkout, and background maintenance have already been upstreamed and
removed from Scalar proper. This patch series provides a C-based
implementation of the final remaining portions of the Scalar command. This
will make it easier for users to experiment with the Scalar command. It will
also make it substantially easier to experiment with moving functionality
from Scalar into core Git, while maintaining backwards-compatibility for
existing Scalar users.
The C-based Scalar has been shipped to Scalar users, and can be tested by
any interested reader:
https://github.com/microsoft/git/releases/tag/v2.33.0.vfs.0.0 (it offers a
Git for Windows installer, a macOS package and an Ubuntu package).
Opportunities
=============
Apart from providing the Scalar command, this contribution is intended to
serve as a basis for further mailing list discussions on moving (some of)
these key concepts into the main Git commands.
For example, we previously discussed the idea of a "git big-clone" that does
much of what "scalar clone" is doing. This patch series is a step to make
such functionality exist in the Git code base while we simmer on what such a
"git big-clone" command-line interface would look like.
This is one of many possible ways to do this. Creating a 'git big-clone'
could lock Git into backwards compatibility concerns so it is necessary to
approach such an endeavor with caution. As a discussion starter, the scalar
clone <url> command does roughly this:
1. git clone --sparse --filter=blob:none /src
2. git -C /src sparse-checkout init --cone
3. git -C /src config (many times)
4. git -C /src maintenance start
It is my hope inspire discussions about what parts of Scalar could go into
core Git, and where, and in which form. While we wish to maintain
backwards-compatibility of Scalar's command-line interface (because it is
already in use), by having the Scalar code in the same code base as Git's,
it will be much easier to move functionality without having to maintain
loose version coupling between independently-versioned Scalar and Git. The
tight version-coupling, along with having access to libgit.a also allows the
C-based implementation of Scalar to be much smaller than the original .NET
version.
For example, we might choose in the future to implement, say, git clone
--scale=partial,cone to initialize a partial clone with a cone-sparse
checkout, that would not only be totally doable, and not only would we
already have precedent and data to prove that this actually makes engineers
happy who have to work on ginormous repositories, but we could then also
implement it by moving parts of contrib/scalar/ to builtin/ (where
contrib/scalar/ would then call the built-ins accordingly rather than
hard-coding the defaults itself).
We now also have the opportunity to discuss the merits of Scalar's clone
caching, which is not actually part of this patch series because it is a bit
coupled with the GVFS parts of microsoft/git for the moment, where clones
automatically get registered with a populated alternate repository that is
identified by the URL, meaning: subsequent clones of the same repository are
vastly faster than the first one because they do not actually download the
already-received objects again, they access the cache instead.
Another thing that I could imagine to be discussed at length is the
distinction between enlistment and worktree (where the latter is the actual
Git worktree and usually lives in the src/ subdirectory of the former). This
encourages untracked and ignored files to be placed outside the worktree,
making Git's job much easier. This idea, too, might find its way in one way
or another into Git proper.
These are just a few concepts in Scalar that do not yet have equivalents in
Git. By putting this initial implementation into contrib/, we create a
foundation for future discussions of these concepts.
We plan on updating the recommended config settings in scalar register as
new Git features are available (such as builtin FSMonitor and sparse-index,
when ready). To facilitate upgrading existing Scalar enlistments, their
paths are automatically added to the [scalar] section of the global Git
config, and the scalar reconfigure --all command will process all of them.
Epilogue
========
Now, to address some questions that I imagine every reader has who made it
this far:
* Why not put the Scalar functionality directly into a built-in? Creating a
Git builtin requires scrutiny over every aspect of the feature, which is
difficult to do while also maintaining the command-line interface
contract and expected behavior of the Scalar command (there are existing
users, after all). By having the Scalar command in contrib/, we present a
simple option for users to have these features in the short term while
the Git contributor community decides which bits to absorb into Git
built-ins.
* Why implement the Scalar command in the Git codebase? We ported Scalar to
the microsoft/git fork for several reasons. First, we realized it was
possible now that the core features exist inside Git itself. Second,
compiling Scalar directly within a version of Git allows us to remove a
version compatibility check from each config option that might or might
not apply based on the installed Git version. Finally, this new location
has greatly simplified our release process and the installation process
for users. We now have ways to install Scalar with microsoft/git via
winget, brew, and apt-get. This has been the case since we shipped
v2.32.0 to our users, read: this setup has served us well already.
* Why contribute Scalar to the Git project? We are biased, of course, yet
we do have evidence that the Scalar command is a helpful tool that offers
an simple way to handle huge repositories with ease. By contributing it
to the core Git project, we are able to share it with more users,
especially some users who do not want to install the microsoft/git fork.
We intend to include Scalar as a component in git-for-windows/git, but
are contributing it here first. Further, we think there is benefit to the
Git developer community as this presents an example of how to set certain
defaults that work for large repositories.
* Does this integrate with the built-in FSMonitor yet? No, not yet. I do
have a couple of add-on patch series lined up, one of them being the
integration with the built-in FSMonitor, which obviously has to wait
until the FSMonitor patch series advances further.
Derrick Stolee (4):
scalar: 'register' sets recommended config and starts maintenance
scalar: 'unregister' stops background maintenance
scalar: implement 'scalar list'
scalar: implement the `run` command
Johannes Schindelin (10):
scalar: create a rudimentary executable
scalar: start documenting the command
scalar: create test infrastructure
scalar: let 'unregister' handle a deleted enlistment directory
gracefully
scalar: implement the `clone` subcommand
scalar: teach 'clone' to support the --single-branch option
scalar: allow reconfiguring an existing enlistment
scalar: teach 'reconfigure' to optionally handle all registered
enlistments
scalar: implement the `version` command
scalar: accept -C and -c options before the subcommand
Matthew John Cheetham (1):
scalar: implement the `delete` command
Makefile | 8 +
contrib/scalar/.gitignore | 5 +
contrib/scalar/Makefile | 57 +++
contrib/scalar/scalar.c | 844 +++++++++++++++++++++++++++++++
contrib/scalar/scalar.txt | 152 ++++++
contrib/scalar/t/Makefile | 78 +++
contrib/scalar/t/t9099-scalar.sh | 88 ++++
7 files changed, 1232 insertions(+)
create mode 100644 contrib/scalar/.gitignore
create mode 100644 contrib/scalar/Makefile
create mode 100644 contrib/scalar/scalar.c
create mode 100644 contrib/scalar/scalar.txt
create mode 100644 contrib/scalar/t/Makefile
create mode 100755 contrib/scalar/t/t9099-scalar.sh
base-commit: ebf3c04b262aa27fbb97f8a0156c2347fecafafb
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1005%2Fdscho%2Fscalar-the-beginning-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1005/dscho/scalar-the-beginning-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1005
--
gitgitgadget
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-08-30 21:34:55
From: Johannes Schindelin <redacted>
The idea of Scalar (https://github.com/microsoft/scalar), and before
that, of VFS for Git, has always been to prove that Git _can_ scale, and
to upstream whatever strategies have been demonstrated to help.
With this patch, we start the journey from that C# project to move what
is left to Git's own `contrib/` directory, reimplementing it in pure C,
with the intention to facilitate integrating the functionality into core
Git all while maintaining backwards-compatibility for existing Scalar
users (which will be much easier when both live in the same worktree).
It was always to plan to contribute all of the proven strategies back to
core Git.
For example, while the virtual filesystem provided by VFS for Git helped
the team developing the Windows operating system to move onto Git, while
trying to upstream it we realized that it cannot be done: getting the
virtual filesystem to work (which we only managed to implement fully on
Windows, but not on, say, macOS or Linux), and the required server-side
support for the GVFS protocol, made this not quite feasible.
The Scalar project learned from that and tackled the problem with
different tactics: instead of pretending to Git that the working
directory is fully populated, it _specifically_ teaches Git about
partial clone (which is based on VFS for Git's cache server), about
sparse checkout (which VFS for Git tried to do transparently, in the
file system layer), and regularly runs maintenance tasks to keep the
repository in a healthy state.
With partial clone, sparse checkout and `git maintenance` having been
upstreamed, there is little left that `scalar.exe` does that which
`git.exe` cannot do. One such thing is that `scalar clone <url>` will
automatically set up a partial, sparse clone, and configure
known-helpful settings from the start.
So let's bring this convenience into Git's tree.
The idea here is that you can (optionally) build Scalar via
make -C contrib/scalar/Makefile
This will build the `scalar` executable and put it into the
contrib/scalar/ subdirectory.
The slightly awkward addition of the `contrib/scalar/*` bits to the
top-level `Makefile` are actually really required: we want to link to
`libgit.a`, which means that we will need to use the very same `CFLAGS`
and `LDFLAGS` as the rest of Git.
An early development version of this patch tried to replicate all the
conditional code in `contrib/scalar/Makefile` (e.g. `NO_POLL`) just like
`contrib/svn-fe/Makefile` used to do before it was retired. It turned
out to be quite the whack-a-mole game: the SHA-1-related flags, the
flags enabling/disabling `compat/poll/`, `compat/regex/`,
`compat/win32mmap.c` & friends depending on the current platform... To
put it mildly: it was a major mess.
Instead, this patch makes minimal changes to the top-level `Makefile` so
that the bits in `contrib/scalar/` can be compiled and linked, and
adds a `contrib/scalar/Makefile` that uses the top-level `Makefile` in a
most minimal way to do the actual compiling.
Note: With this commit, we only establish the infrastructure, no
Scalar functionality is implemented yet; We will do that incrementally
over the next few commits.
Signed-off-by: Johannes Schindelin <redacted>
---
Makefile | 8 ++++++++
contrib/scalar/.gitignore | 2 ++
contrib/scalar/Makefile | 34 ++++++++++++++++++++++++++++++++++
contrib/scalar/scalar.c | 36 ++++++++++++++++++++++++++++++++++++
4 files changed, 80 insertions(+)
create mode 100644 contrib/scalar/.gitignore
create mode 100644 contrib/scalar/Makefile
create mode 100644 contrib/scalar/scalar.c
@@ -0,0 +1,34 @@+QUIET_SUBDIR0=+$(MAKE)-C# space to separate -C and subdir+QUIET_SUBDIR1=++ifneq ($(findstring s,$(MAKEFLAGS)),s)+ifndef V+QUIET_SUBDIR0=+@subdir=+QUIET_SUBDIR1=;$(NO_SUBDIR)echo' 'SUBDIR$$subdir;\+$(MAKE)$(PRINT_DIR)-C$$subdir+else+exportV+endif+endif++all:++include ../../config.mak.uname+-include ../../config.mak.autogen+-include ../../config.mak++TARGETS=scalar$(X)scalar.o+GITLIBS=../../common-main.o../../libgit.a../../xdiff/lib.a++all:scalar$X++$(GITLIBS):+$(QUIET_SUBDIR0)../..$(QUIET_SUBDIR1)$(subst../../,,$@)++$(TARGETS):$(GITLIBS)scalar.c+$(QUIET_SUBDIR0)../..$(QUIET_SUBDIR1)$(patsubst%,contrib/scalar/%,$@)++clean:+$(RM)$(TARGETS)++.PHONY:allcleanFORCE
@@ -30,5 +31,16 @@ $(TARGETS): $(GITLIBS) scalar.cclean:$(RM)$(TARGETS)+$(RM)scalar.1scalar.htmlscalar.xml-.PHONY:allcleanFORCE+docs:scalar.htmlscalar.1++scalar.html:|scalar.1 # prevent them from trying to build `doc.dep` in parallel++scalar.html scalar.1:scalar.txt+$(QUIET_SUBDIR0)../../Documentation$(QUIET_SUBDIR1)\+MAN_TXT=../contrib/scalar/scalar.txt\+../contrib/scalar/$@+$(QUIET)testscalar.1!="$@"||mv../../Documentation/$@.++.PHONY:allcleandocsFORCE
@@ -0,0 +1,38 @@+scalar(1)+=========++NAME+----+scalar - an opinionated repository management tool++SYNOPSIS+--------+[verse]+scalar <command> [<options>]++DESCRIPTION+-----------++Scalar is an opinionated repository management tool. By creating new+repositories or registering existing repositories with Scalar, your Git+experience will speed up. Scalar sets advanced Git config settings,+maintains your repositories in the background, and helps reduce data sent+across the network.++An important Scalar concept is the enlistment: this is the top-level directory+of the project. It usually contains the subdirectory `src/` which is a Git+worktree. This encourages the separation between tracked files (inside `src/`)+and untracked files, such as build artifacts (outside `src/`). When registering+an existing Git worktree with Scalar whose name is not `src`, the enlistment+will be identical to the worktree.++The `scalar` command implements various subcommands, and different options+depending on the subcommand.++SEE ALSO+--------+linkgit:git-maintenance[1].++Scalar+---+Associated with the linkgit:git[1] suite
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-08-30 21:34:59
From: Johannes Schindelin <redacted>
To test the Scalar command, create a test script in contrib/scalar/t
that is executed as `make -C contrib/scalar test`. Since Scalar has no
meaningful capabilities yet, the only test is rather simple. We will add
more tests in subsequent commits that introduce corresponding, new
functionality.
Note: this test script is intended to test `scalar` only lightly, even
after all of the functionality is implemented.
A more comprehensive functional (or: integration) test suite can be
found at https://github.com/microsoft/scalar; It is used in the workflow
https://github.com/microsoft/git/blob/HEAD/.github/workflows/scalar-functional-tests.yml
in Microsoft's Git fork. This test suite performs end-to-end tests with
a real remote repository, and is run as part of the regular CI builds.
Since those tests require some functionality supported only by
Microsoft's Git fork ("GVFS protocol"), there is no intention to port
that fuller test suite to `contrib/scalar/`.
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/Makefile | 17 +++++--
contrib/scalar/t/Makefile | 78 ++++++++++++++++++++++++++++++++
contrib/scalar/t/t9099-scalar.sh | 17 +++++++
3 files changed, 109 insertions(+), 3 deletions(-)
create mode 100644 contrib/scalar/t/Makefile
create mode 100755 contrib/scalar/t/t9099-scalar.sh
@@ -21,7 +22,7 @@ include ../../config.mak.unameTARGETS=scalar$(X)scalar.oGITLIBS=../../common-main.o../../libgit.a../../xdiff/lib.a-all:scalar$X+all:scalar$X ../../bin-wrappers/scalar$(GITLIBS):$(QUIET_SUBDIR0)../..$(QUIET_SUBDIR1)$(subst../../,,$@)
@@ -30,9 +31,19 @@ $(TARGETS): $(GITLIBS) scalar.c$(QUIET_SUBDIR0)../..$(QUIET_SUBDIR1)$(patsubst%,contrib/scalar/%,$@)clean:-$(RM)$(TARGETS)+$(RM)$(TARGETS)../../bin-wrappers/scalar$(RM)scalar.1scalar.htmlscalar.xml+../../bin-wrappers/scalar:../../wrap-for-bin.shMakefile+@mkdir-p../../bin-wrappers+$(QUIET_GEN)sed-e'1s|#!.*/sh|#!$(SHELL_PATH_SQ)|'\+-e's|@@BUILD_DIR@@|$(shell cd ../.. && pwd)|'\+-e's|@@PROG@@|contrib/scalar/scalar$(X)|'<$<>$@&&\+chmod+x$@++test:all+$(MAKE)-Ct+docs:scalar.htmlscalar.1scalar.html:|scalar.1 # prevent them from trying to build `doc.dep` in parallel
@@ -0,0 +1,17 @@+#!/bin/sh++test_description='test the `scalar` command'++TEST_DIRECTORY=$PWD/../../../t+exportTEST_DIRECTORY++# Make it work with --no-bin-wrappers+PATH=$PWD/..:$PATH++.../../../t/test-lib.sh++test_expect_success'scalar shows a usage''+test_expect_code129scalar-h+'++test_done
From: Derrick Stolee via GitGitGadget <hidden> Date: 2021-08-30 21:35:00
From: Derrick Stolee <redacted>
Let's start implementing the `register` command. With this commit,
recommended settings are configured upon `scalar register`, and Git's
background maintenance is started.
The recommended config settings may very well change in the future. For
example, once the built-in FSMonitor is available, we will want to
enable it upon `scalar register`. For that reason, we explicitly support
running `scalar register` in an already-registered enlistment.
Co-authored-by: Victoria Dye [off-list ref]
Signed-off-by: Derrick Stolee <redacted>
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 256 ++++++++++++++++++++++++++++++++++++++
contrib/scalar/scalar.txt | 14 ++-
2 files changed, 269 insertions(+), 1 deletion(-)
@@ -5,11 +5,267 @@#include"cache.h"#include"gettext.h"#include"parse-options.h"+#include"config.h"+#include"run-command.h"++/*+*Removethedeepestsubdirectoryintheprovidedpathstring.Pathmustnot+*includeatrailingpathseparator.Returns1ifparentdirectoryfound,+*otherwise0.+*/+staticintstrbuf_parent_directory(structstrbuf*buf)+{+size_tlen=buf->len;+size_toffset=offset_1st_component(buf->buf);+char*path_sep=find_last_dir_sep(buf->buf+offset);+strbuf_setlen(buf,path_sep?path_sep-buf->buf:offset);++returnbuf->len<len;+}++staticvoidsetup_enlistment_directory(intargc,constchar**argv,+constchar*const*usagestr,+conststructoption*options,+structstrbuf*enlistment_root)+{+structstrbufpath=STRBUF_INIT;+char*root;+intenlistment_found=0;++if(startup_info->have_repository)+BUG("gitdir already set up?!?");++if(argc>1)+usage_with_options(usagestr,options);++/* find the worktree, determine its corresponding root */+if(argc==1)+strbuf_add_absolute_path(&path,argv[0]);+elseif(strbuf_getcwd(&path)<0)+die(_("need a working directory"));++strbuf_trim_trailing_dir_sep(&path);+do{+constsize_tlen=path.len;++/* check if currently in enlistment root with src/ workdir */+strbuf_addstr(&path,"/src/.git");+if(is_git_directory(path.buf)){+strbuf_strip_suffix(&path,"/.git");++if(enlistment_root)+strbuf_add(enlistment_root,path.buf,len);++enlistment_found=1;+break;+}++/* reset to original path */+strbuf_setlen(&path,len);++/* check if currently in workdir */+strbuf_addstr(&path,"/.git");+if(is_git_directory(path.buf)){+strbuf_setlen(&path,len);++if(enlistment_root){+/*+*Iftheworktree'sdirectory'snameis`src`,theenlistmentisthe+*parentdirectory,otherwiseitisidenticaltotheworktree.+*/+root=strip_path_suffix(path.buf,"src");+strbuf_addstr(enlistment_root,root?root:path.buf);+free(root);+}++enlistment_found=1;+break;+}++strbuf_setlen(&path,len);+}while(strbuf_parent_directory(&path));++if(!enlistment_found)+die(_("could not find enlistment root"));++if(chdir(path.buf)<0)+die_errno(_("could not switch to '%s'"),path.buf);++strbuf_release(&path);+setup_git_directory();+}++staticintrun_git(constchar*arg,...)+{+structstrvecargv=STRVEC_INIT;+va_listargs;+constchar*p;+intres;++va_start(args,arg);+strvec_push(&argv,arg);+while((p=va_arg(args,constchar*)))+strvec_push(&argv,p);+va_end(args);++res=run_command_v_opt(argv.v,RUN_GIT_CMD);++strvec_clear(&argv);+returnres;+}++staticintset_recommended_config(void)+{+struct{+constchar*key;+constchar*value;+}config[]={+{"am.keepCR","true"},+{"core.FSCache","true"},+{"core.multiPackIndex","true"},+{"core.preloadIndex","true"},+#ifndef WIN32+{"core.untrackedCache","true"},+#else+/*+*Unfortunately,Scalar'sFunctionalTestsdemonstrated+*thattheuntrackedcachefeatureisunreliableonWindows+*(whichisabummerbecausethatplatformwouldbenefitthe+*mostfromit).Forsomereason,freshlycreatedfilesseem+*nottoupdatethedirectory's`lastModified`time+*immediately,buttheuntrackedcachewouldneedtorelyon+*that.+*+*Therefore,withasadheart,wedisablethisveryuseful+*featureonWindows.+*/+{"core.untrackedCache","false"},+#endif+{"core.bare","false"},+{"core.logAllRefUpdates","true"},+{"credential.https://dev.azure.com.useHttpPath","true"},+{"credential.validate","false"},/* GCM4W-only */+{"gc.auto","0"},+{"gui.GCWarning","false"},+{"index.threads","true"},+{"index.version","4"},+{"merge.stat","false"},+{"merge.renames","false"},+{"pack.useBitmaps","false"},+{"pack.useSparse","true"},+{"receive.autoGC","false"},+{"reset.quiet","true"},+{"feature.manyFiles","false"},+{"feature.experimental","false"},+{"fetch.unpackLimit","1"},+{"fetch.writeCommitGraph","false"},+#ifdef WIN32+{"http.sslBackend","schannel"},+#endif+{"status.aheadBehind","false"},+{"commitGraph.generationVersion","1"},+{"core.autoCRLF","false"},+{"core.safeCRLF","false"},+{NULL,NULL},+};+inti;+char*value;++for(i=0;config[i].key;i++){+if(git_config_get_string(config[i].key,&value)){+trace2_data_string("scalar",the_repository,config[i].key,"created");+if(git_config_set_gently(config[i].key,+config[i].value)<0)+returnerror(_("could not configure %s=%s"),+config[i].key,config[i].value);+}else{+trace2_data_string("scalar",the_repository,config[i].key,"exists");+free(value);+}+}++/*+*The`log.excludeDecoration`settingisspecialbecauseitallows+*formultiplevalues.+*/+if(git_config_get_string("log.excludeDecoration",&value)){+trace2_data_string("scalar",the_repository,+"log.excludeDecoration","created");+if(git_config_set_multivar_gently("log.excludeDecoration",+"refs/prefetch/*",+CONFIG_REGEX_NONE,0))+returnerror(_("could not configure "+"log.excludeDecoration"));+}else{+trace2_data_string("scalar",the_repository,+"log.excludeDecoration","exists");+free(value);+}++return0;+}++staticintstart_maintenance(void)+{+returnrun_git("maintenance","start",NULL);+}++staticintadd_enlistment(void)+{+intres;++if(!the_repository->worktree)+die(_("Scalar enlistments require a worktree"));++res=run_git("config","--global","--get","--fixed-value",+"scalar.repo",the_repository->worktree,NULL);++/*+*Ifthesettingisalreadythere,thendonothing.+*/+if(!res)+return0;++returnrun_git("config","--global","--add",+"scalar.repo",the_repository->worktree,NULL);+}++staticintregister_dir(void)+{+intres=add_enlistment();++if(!res)+res=set_recommended_config();++if(!res)+res=start_maintenance();++returnres;+}++staticintcmd_register(intargc,constchar**argv)+{+structoptionoptions[]={+OPT_END(),+};+constchar*constusage[]={+N_("scalar register [<enlistment>]"),+NULL+};++argc=parse_options(argc,argv,NULL,options,+usage,0);++setup_enlistment_directory(argc,argv,usage,options,NULL);++returnregister_dir();+}staticstruct{constchar*name;int(*fn)(int,constchar**);}builtins[]={+{"register",cmd_register},{NULL,NULL},};
@@ -29,6 +29,18 @@ will be identical to the worktree. The `scalar` command implements various subcommands, and different options depending on the subcommand.+COMMANDS+--------++Register+~~~~~~~~++register [<enlistment>]::+ Adds the enlistment's repository to the list of registered repositories+ and starts background maintenance. If `<enlistment>` is not provided,+ then the enlistment associated with the current working directory is+ registered.+ SEE ALSO -------- linkgit:git-maintenance[1].
From: Derrick Stolee via GitGitGadget <hidden> Date: 2021-08-30 21:35:01
From: Derrick Stolee <redacted>
Just like `scalar register` starts the scheduled background maintenance,
`scalar unregister` stops it. Note that we use `git maintenance start`
in `scalar register`, but we do not use `git maintenance stop` in
`scalar unregister`: this would stop maintenance for _all_ repositories,
not just for the one we want to unregister.
The `unregister` command also removes the corresponding entry from the
`[scalar]` section in the global Git config.
Co-authored-by: Victoria Dye [off-list ref]
Signed-off-by: Derrick Stolee <redacted>
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 50 ++++++++++++++++++++++++++++++++-------
contrib/scalar/scalar.txt | 8 +++++++
2 files changed, 50 insertions(+), 8 deletions(-)
@@ -41,6 +42,13 @@ register [<enlistment>]:: then the enlistment associated with the current working directory is registered.+Unregister+~~~~~~~~~~++unregister [<enlistment>]::+ Remove the specified repository from the list of repositories+ registered with Scalar and stop the scheduled background maintenance.+ SEE ALSO -------- linkgit:git-maintenance[1].
From: Derrick Stolee via GitGitGadget <hidden> Date: 2021-08-30 21:35:03
From: Derrick Stolee <redacted>
The produced list simply consists of those repositories registered under
the multi-valued `scalar.repo` config setting in the user's Git config.
Signed-off-by: Derrick Stolee <redacted>
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 11 +++++++++++
contrib/scalar/scalar.txt | 12 +++++++++++-
2 files changed, 22 insertions(+), 1 deletion(-)
@@ -258,6 +258,16 @@ static int unregister_dir(void)returnres;}+staticintcmd_list(intargc,constchar**argv)+{+if(argc!=1)+die(_("`scalar list` does not take arguments"));++if(run_git("config","--global","--get-all","scalar.repo",NULL)<0)+return-1;+return0;+}+staticintcmd_register(intargc,constchar**argv){structoptionoptions[]={
@@ -28,11 +29,20 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment will be identical to the worktree. The `scalar` command implements various subcommands, and different options-depending on the subcommand.+depending on the subcommand. With the exception of `list`, all subcommands+expect to be run in an enlistment. COMMANDS --------+List+~~~~++list::+ To see which repositories are currently registered by the service, run+ `scalar list`. This subcommand does not need to be run inside a Scalar+ enlistment.+ Register ~~~~~~~~
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-08-30 21:35:04
From: Johannes Schindelin <redacted>
When a user deleted an enlistment manually, let's be generous and
_still_ unregister it.
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 46 ++++++++++++++++++++++++++++++++
contrib/scalar/t/t9099-scalar.sh | 15 +++++++++++
2 files changed, 61 insertions(+)
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-08-30 21:35:06
From: Johannes Schindelin <redacted>
This implements Scalar's opinionated `clone` command: it tries to use a
partial clone and sets up a sparse checkout by default. In contrast to
`git clone`, `scalar clone` sets up the worktree in the `src/`
subdirectory, to encourage a separation between the source files and the
build output (which helps Git tremendously because it avoids untracked
files that have to be specifically ignored when refreshing the index).
Also, it registers the repository for regular, scheduled maintenance,
and configures a flurry of configuration settings based on the
experience and experiments of the Microsoft Windows and the Microsoft
Office development teams.
Note: We intentionally use a slightly wasteful `set_config()` function
(which does not reuse a single `strbuf`, for example, though performance
_really_ does not matter here) for convenience and readability.
Also note: since the `scalar clone` command is by far the most commonly
called `scalar` subcommand, we document it at the top of the manual
page.
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 200 +++++++++++++++++++++++++++++++
contrib/scalar/scalar.txt | 35 +++++-
contrib/scalar/t/t9099-scalar.sh | 32 +++++
3 files changed, 262 insertions(+), 5 deletions(-)
@@ -258,6 +259,204 @@ static int unregister_dir(void)returnres;}+/* printf-style interface, expects `<key>=<value>` argument */+staticintset_config(constchar*fmt,...)+{+structstrbufbuf=STRBUF_INIT;+char*value;+intres;+va_listargs;++va_start(args,fmt);+strbuf_vaddf(&buf,fmt,args);+va_end(args);++value=strchr(buf.buf,'=');+if(value)+*(value++)='\0';+res=git_config_set_gently(buf.buf,value);+strbuf_release(&buf);++returnres;+}++staticchar*remote_default_branch(constchar*url)+{+structchild_processcp=CHILD_PROCESS_INIT;+structstrbufout=STRBUF_INIT;++cp.git_cmd=1;+strvec_pushl(&cp.args,"ls-remote","--symref",url,"HEAD",NULL);+strbuf_addstr(&out,"-\n");+if(!pipe_command(&cp,NULL,0,&out,0,NULL,0)){+char*ref=out.buf;++while((ref=strstr(ref+1,"\nref: "))){+constchar*p;+char*head,*branch;++ref+=strlen("\nref: ");+head=strstr(ref,"\tHEAD");++if(!head||memchr(ref,'\n',head-ref))+continue;++if(skip_prefix(ref,"refs/heads/",&p)){+branch=xstrndup(p,head-p);+strbuf_release(&out);+returnbranch;+}++error(_("remote HEAD is not a branch: '%.*s'"),+(int)(head-ref),ref);+strbuf_release(&out);+returnNULL;+}+}+warning(_("failed to get default branch name from remote; "+"using local default"));+strbuf_reset(&out);++child_process_init(&cp);+cp.git_cmd=1;+strvec_pushl(&cp.args,"symbolic-ref","--short","HEAD",NULL);+if(!pipe_command(&cp,NULL,0,&out,0,NULL,0)){+strbuf_trim(&out);+returnstrbuf_detach(&out,NULL);+}++strbuf_release(&out);+error(_("failed to get default branch name"));+returnNULL;+}++staticintcmd_clone(intargc,constchar**argv)+{+constchar*branch=NULL;+intfull_clone=0;+structoptionclone_options[]={+OPT_STRING('b',"branch",&branch,N_("<branch>"),+N_("branch to checkout after clone")),+OPT_BOOL(0,"full-clone",&full_clone,+N_("when cloning, create full working directory")),+OPT_END(),+};+constchar*constclone_usage[]={+N_("scalar clone [<options>] [--] <repo> [<dir>]"),+NULL+};+constchar*url;+char*enlistment=NULL,*dir=NULL;+structstrbufbuf=STRBUF_INIT;+intres;++argc=parse_options(argc,argv,NULL,clone_options,clone_usage,0);++if(argc==2){+url=argv[0];+enlistment=xstrdup(argv[1]);+}elseif(argc==1){+url=argv[0];++strbuf_addstr(&buf,url);+/* Strip trailing slashes, if any */+while(buf.len>0&&is_dir_sep(buf.buf[buf.len-1]))+strbuf_setlen(&buf,buf.len-1);+/* Strip suffix `.git`, if any */+strbuf_strip_suffix(&buf,".git");++enlistment=find_last_dir_sep(buf.buf);+if(!enlistment){+die(_("cannot deduce worktree name from '%s'"),url);+}+enlistment=xstrdup(enlistment+1);+}else{+usage_msg_opt(_("You must specify a repository to clone."),+clone_usage,clone_options);+}++if(is_directory(enlistment))+die(_("directory '%s' exists already"),enlistment);++dir=xstrfmt("%s/src",enlistment);++strbuf_reset(&buf);+if(branch)+strbuf_addf(&buf,"init.defaultBranch=%s",branch);+else{+char*b=repo_default_branch_name(the_repository,1);+strbuf_addf(&buf,"init.defaultBranch=%s",b);+free(b);+}++if((res=run_git("-c",buf.buf,"init","--",dir,NULL)))+gotocleanup;++if(chdir(dir)<0){+res=error_errno(_("could not switch to '%s'"),dir);+gotocleanup;+}++setup_git_directory();++/* common-main already logs `argv` */+trace2_def_repo(the_repository);++if(!branch&&!(branch=remote_default_branch(url))){+res=error(_("failed to get default branch for '%s'"),url);+gotocleanup;+}++if(set_config("remote.origin.url=%s",url)||+set_config("remote.origin.fetch="+"+refs/heads/*:refs/remotes/origin/*")||+set_config("remote.origin.promisor=true")||+set_config("remote.origin.partialCloneFilter=blob:none")){+res=error(_("could not configure remote in '%s'"),dir);+gotocleanup;+}++if(!full_clone&&+(res=run_git("sparse-checkout","init","--cone",NULL)))+gotocleanup;++if(set_recommended_config())+returnerror(_("could not configure '%s'"),dir);++if((res=run_git("fetch","--quiet","origin",NULL))){+warning(_("partial clone failed; attempting full clone"));++if(set_config("remote.origin.promisor")||+set_config("remote.origin.partialCloneFilter")){+res=error(_("could not configure for full clone"));+gotocleanup;+}++if((res=run_git("fetch","--quiet","origin",NULL)))+gotocleanup;+}++if((res=set_config("branch.%s.remote=origin",branch)))+gotocleanup;+if((res=set_config("branch.%s.merge=refs/heads/%s",+branch,branch)))+gotocleanup;++strbuf_reset(&buf);+strbuf_addf(&buf,"origin/%s",branch);+res=run_git("checkout","-f","-t",buf.buf,NULL);+if(res)+gotocleanup;++res=register_dir();++cleanup:+free(enlistment);+free(dir);+strbuf_release(&buf);+returnres;+}+staticintcmd_list(intargc,constchar**argv){if(argc!=1)
@@ -29,19 +30,43 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment will be identical to the worktree. The `scalar` command implements various subcommands, and different options-depending on the subcommand. With the exception of `list`, all subcommands-expect to be run in an enlistment.+depending on the subcommand. With the exception of `clone` and `list`, all+subcommands expect to be run in an enlistment. COMMANDS --------+Clone+~~~~~++clone [<options>] <url> [<enlistment>]::+ Clones the specified repository, similar to linkgit:git-clone[1]. By+ default, only commit and tree objects are cloned. Once finished, the+ worktree is located at `<enlistment>/src`.+++The sparse-checkout feature is enabled (except when run with `--full-clone`)+and the only files present are those in the top-level directory. Use+`git sparse-checkout set` to expand the set of directories you want to see,+or `git sparse-checkout disable` to expand to all files (see+linkgit:git-sparse-checkout[1] for more details). You can explore the+subdirectories outside your sparse-checkout by using `git ls-tree HEAD`.++-b <name>::+--branch <name>::+ Instead of checking out the branch pointed to by the cloned repository's+ HEAD, check out the `<name>` branch instead.++--[no-]full-clone::+ A sparse-checkout is initialized by default. This behavior can be turned+ off via `--full-clone`.+ List ~~~~ list:: To see which repositories are currently registered by the service, run- `scalar list`. This subcommand does not need to be run inside a Scalar- enlistment.+ `scalar list`. This subcommand, like `clone`, does not need to be run+ inside a Scalar enlistment. Register ~~~~~~~~
@@ -61,7 +86,7 @@ unregister [<enlistment>]:: SEE ALSO ---------linkgit:git-maintenance[1].+linkgit:git-clone[1], linkgit:git-maintenance[1]. Scalar ---
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-08-30 21:35:07
From: Johannes Schindelin <redacted>
Just like `git clone`, the `scalar clone` command now also offers to
restrict the clone to a single branch.
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 9 +++++++--
contrib/scalar/scalar.txt | 12 +++++++++++-
contrib/scalar/t/t9099-scalar.sh | 6 +++++-
3 files changed, 23 insertions(+), 4 deletions(-)
@@ -333,12 +333,15 @@ static char *remote_default_branch(const char *url)staticintcmd_clone(intargc,constchar**argv){constchar*branch=NULL;-intfull_clone=0;+intfull_clone=0,single_branch=0;structoptionclone_options[]={OPT_STRING('b',"branch",&branch,N_("<branch>"),N_("branch to checkout after clone")),OPT_BOOL(0,"full-clone",&full_clone,N_("when cloning, create full working directory")),+OPT_BOOL(0,"single-branch",&single_branch,+N_("only download metadata for the branch that will "+"be checked out")),OPT_END(),};constchar*constclone_usage[]={
@@ -409,7 +412,9 @@ static int cmd_clone(int argc, const char **argv)if(set_config("remote.origin.url=%s",url)||set_config("remote.origin.fetch="-"+refs/heads/*:refs/remotes/origin/*")||+"+refs/heads/%s:refs/remotes/origin/%s",+single_branch?branch:"*",+single_branch?branch:"*")||set_config("remote.origin.promisor=true")||set_config("remote.origin.partialCloneFilter=blob:none")){res=error(_("could not configure remote in '%s'"),dir);
@@ -56,6 +56,16 @@ subdirectories outside your sparse-checkout by using `git ls-tree HEAD`. Instead of checking out the branch pointed to by the cloned repository's HEAD, check out the `<name>` branch instead.+--[no-]single-branch::+ Clone only the history leading to the tip of a single branch,+ either specified by the `--branch` option or the primary+ branch remote's `HEAD` points at.+++Further fetches into the resulting repository will only update the+remote-tracking branch for the branch this option was used for the initial+cloning. If the HEAD at the remote did not point at any branch when+`--single-branch` clone was made, no remote-tracking branch is created.+ --[no-]full-clone:: A sparse-checkout is initialized by default. This behavior can be turned off via `--full-clone`.
From: Derrick Stolee via GitGitGadget <hidden> Date: 2021-08-30 21:35:09
From: Derrick Stolee <redacted>
Note: this subcommand is provided primarily for backwards-compatibility,
for existing Scalar uses. It is mostly just a shim for `git
maintenance`, mapping task names from the way Scalar called them to the
way Git calls them.
The reason why those names differ? The background maintenance was first
implemented in Scalar, and when it was contributed as a patch series
implementing the `git maintenance` command, reviewers suggested better
names, those suggestions were accepted before the patches were
integrated into core Git.
Signed-off-by: Derrick Stolee <redacted>
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 64 +++++++++++++++++++++++++++++++++++++++
contrib/scalar/scalar.txt | 19 ++++++++++++
2 files changed, 83 insertions(+)
@@ -490,6 +490,69 @@ static int cmd_register(int argc, const char **argv)returnregister_dir();}+staticintcmd_run(intargc,constchar**argv)+{+structoptionoptions[]={+OPT_END(),+};+struct{+constchar*arg,*task;+}tasks[]={+{"config",NULL},+{"commit-graph","commit-graph"},+{"fetch","prefetch"},+{"loose-objects","loose-objects"},+{"pack-files","incremental-repack"},+{NULL,NULL}+};+structstrbufbuf=STRBUF_INIT;+constchar*usagestr[]={NULL,NULL};+inti;++strbuf_addstr(&buf,N_("scalar run <task> [<enlistment>]\nTasks:\n"));+for(i=0;tasks[i].arg;i++)+strbuf_addf(&buf,"\t%s\n",tasks[i].arg);+usagestr[0]=buf.buf;++argc=parse_options(argc,argv,NULL,options,+usagestr,0);++if(argc==0)+usage_with_options(usagestr,options);++if(!strcmp("all",argv[0]))+i=-1;+else{+for(i=0;tasks[i].arg&&strcmp(tasks[i].arg,argv[0]);i++)+;/* keep looking for the task */++if(i>0&&!tasks[i].arg){+error(_("no such task: '%s'"),argv[0]);+usage_with_options(usagestr,options);+}+}++argc--;+argv++;+setup_enlistment_directory(argc,argv,usagestr,options,NULL);+strbuf_release(&buf);++if(i==0)+returnregister_dir();++if(i>0)+returnrun_git("maintenance","run",+"--task",tasks[i].task,NULL);++if(register_dir())+return-1;+for(i=1;tasks[i].arg;i++)+if(run_git("maintenance","run",+"--task",tasks[i].task,NULL))+return-1;+return0;+}+staticintremove_deleted_enlistment(structstrbuf*path){intres=0;
@@ -94,6 +95,24 @@ unregister [<enlistment>]:: Remove the specified repository from the list of repositories registered with Scalar and stop the scheduled background maintenance.+Run+~~~++scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]::+ Run the given maintenance task (or all tasks, if `all` was specified).+ Except for `all` and `config`, this subcommand simply hands off to+ linkgit:git-maintenance[1] (mapping `fetch` to `prefetch` and+ `pack-files` to `incremental-repack`).+++These tasks are run automatically as part of the scheduled maintenance,+as soon as the repository is registered with Scalar. It should therefore+not be necessary to run this subcommand manually.+++The `config` task is specific to Scalar and configures all those+opinionated default settings that make Git work more efficiently with+large repositories. As this task is run as part of `scalar clone`+automatically, explicit invocations of this task are rarely needed.+ SEE ALSO -------- linkgit:git-clone[1], linkgit:git-maintenance[1].
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-08-30 21:35:11
From: Johannes Schindelin <redacted>
This comes in handy during Scalar upgrades, or when config settings were
messed up by mistake.
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 81 ++++++++++++++++++++------------
contrib/scalar/scalar.txt | 8 ++++
contrib/scalar/t/t9099-scalar.sh | 8 ++++
3 files changed, 68 insertions(+), 29 deletions(-)
@@ -113,6 +114,13 @@ opinionated default settings that make Git work more efficiently with large repositories. As this task is run as part of `scalar clone` automatically, explicit invocations of this task are rarely needed.+Reconfigure+~~~~~~~~~~~++After a Scalar upgrade, or when the configuration of a Scalar enlistment+was somehow corrupted or changed by mistake, this subcommand allows to+reconfigure the enlistment.+ SEE ALSO -------- linkgit:git-clone[1], linkgit:git-maintenance[1].
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-08-30 21:35:12
From: Johannes Schindelin <redacted>
After a Scalar upgrade, it can come in really handy if there is an easy
way to reconfigure all Scalar enlistments. This new option offers this
functionality.
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 61 ++++++++++++++++++++++++++++++--
contrib/scalar/scalar.txt | 10 ++++--
contrib/scalar/t/t9099-scalar.sh | 3 ++
3 files changed, 68 insertions(+), 6 deletions(-)
@@ -494,22 +494,77 @@ static int cmd_register(int argc, const char **argv)returnregister_dir();}+staticintget_scalar_repos(constchar*key,constchar*value,void*data)+{+structstring_list*list=data;++if(!strcmp(key,"scalar.repo"))+string_list_append(list,value);++return0;+}+staticintcmd_reconfigure(intargc,constchar**argv){+intall=0;structoptionoptions[]={+OPT_BOOL('a',"all",&all,+N_("reconfigure all registered enlistments")),OPT_END(),};constchar*constusage[]={-N_("scalar reconfigure [<enlistment>]"),+N_("scalar reconfigure [--all | <enlistment>]"),NULL};+structstring_listscalar_repos=STRING_LIST_INIT_DUP;+inti,res=0;+structrepositoryr={NULL};+structstrbufcommondir=STRBUF_INIT,gitdir=STRBUF_INIT;argc=parse_options(argc,argv,NULL,options,usage,0);-setup_enlistment_directory(argc,argv,usage,options,NULL);+if(!all){+setup_enlistment_directory(argc,argv,usage,options,NULL);++returnset_recommended_config(1);+}++if(argc>0)+usage_msg_opt(_("--all or <enlistment>, but not both"),+usage,options);++git_config(get_scalar_repos,&scalar_repos);-returnset_recommended_config(1);+for(i=0;i<scalar_repos.nr;i++){+constchar*dir=scalar_repos.items[i].string;++strbuf_reset(&commondir);+strbuf_reset(&gitdir);++if(chdir(dir)<0){+warning_errno(_("could not switch to '%s'"),dir);+res=-1;+}elseif(discover_git_directory(&commondir,&gitdir)<0){+warning_errno(_("git repository gone in '%s'"),dir);+res=-1;+}else{+git_config_clear();++the_repository=&r;+r.commondir=commondir.buf;+r.gitdir=gitdir.buf;++if(set_recommended_config(1)<0)+res=-1;+}+}++string_list_clear(&scalar_repos,1);+strbuf_release(&commondir);+strbuf_release(&gitdir);++returnres;}staticintcmd_run(intargc,constchar**argv)
@@ -32,8 +32,8 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment will be identical to the worktree. The `scalar` command implements various subcommands, and different options-depending on the subcommand. With the exception of `clone` and `list`, all-subcommands expect to be run in an enlistment.+depending on the subcommand. With the exception of `clone`, `list` and+`reconfigure --all`, all subcommands expect to be run in an enlistment. COMMANDS --------
@@ -121,6 +121,10 @@ After a Scalar upgrade, or when the configuration of a Scalar enlistment was somehow corrupted or changed by mistake, this subcommand allows to reconfigure the enlistment.+With the `--all` option, all enlistments currently registered with Scalar+will be reconfigured. This option is meant to to be run every time Scalar+was upgraded.+ SEE ALSO -------- linkgit:git-clone[1], linkgit:git-maintenance[1].
From: Matthew John Cheetham via GitGitGadget <hidden> Date: 2021-08-30 21:35:13
From: Matthew John Cheetham <redacted>
Delete an enlistment by first unregistering the repository and then
deleting the enlistment directory (usually the directory containing the
worktree `src/` directory).
On Windows, if the current directory is inside the enlistment's
directory, change to the parent of the enlistment directory, to allow us
to delete the enlistment (directories used by processes e.g. as current
working directories cannot be deleted on Windows).
Co-authored-by: Victoria Dye [off-list ref]
Signed-off-by: Matthew John Cheetham <redacted>
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 55 ++++++++++++++++++++++++++++++++
contrib/scalar/scalar.txt | 8 +++++
contrib/scalar/t/t9099-scalar.sh | 9 ++++++
3 files changed, 72 insertions(+)
@@ -125,6 +126,13 @@ With the `--all` option, all enlistments currently registered with Scalar will be reconfigured. This option is meant to to be run every time Scalar was upgraded.+Delete+~~~~~~++delete <enlistment>::+ This subcommand lets you delete an existing Scalar enlistment from your+ local file system, unregistering the repository.+ SEE ALSO -------- linkgit:git-clone[1], linkgit:git-maintenance[1].
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-08-30 21:35:16
From: Johannes Schindelin <redacted>
The `git` executable has these two very useful options:
-C <directory>:
switch to the specified directory before performing any actions
-c <key>=<value>:
temporarily configure this setting for the duration of the
specified scalar subcommand
With this commit, we teach the `scalar` executable the same trick.
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 22 +++++++++++++++++++++-
contrib/scalar/scalar.txt | 10 ++++++++++
2 files changed, 31 insertions(+), 1 deletion(-)
@@ -806,6 +806,25 @@ int cmd_main(int argc, const char **argv)structstrbufscalar_usage=STRBUF_INIT;inti;+while(argc>1&&*argv[1]=='-'){+if(!strcmp(argv[1],"-C")){+if(argc<3)+die(_("-C requires a <directory>"));+if(chdir(argv[2])<0)+die_errno(_("could not change to '%s'"),+argv[2]);+argc-=2;+argv+=2;+}elseif(!strcmp(argv[1],"-c")){+if(argc<3)+die(_("-c requires a <key>=<value> argument"));+git_config_push_parameter(argv[2]);+argc-=2;+argv+=2;+}else+break;+}+if(argc>1){argv++;argc--;
@@ -36,6 +36,16 @@ The `scalar` command implements various subcommands, and different options depending on the subcommand. With the exception of `clone`, `list` and `reconfigure --all`, all subcommands expect to be run in an enlistment.+The following options can be specified _before_ the subcommand:++-C <directory>::+ Before running the subcommand, change the working directory. This+ option imitates the same option of linkgit:git[1].++-c <key>=<value>::+ For the duration of running the specified subcommand, configure this+ setting. This option imitates the same option of linkgit:git[1].+ COMMANDS --------
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-08-30 21:35:17
From: Johannes Schindelin <redacted>
The .NET version of Scalar has a `version` command. This was necessary
because it was versioned independently of Git.
Since Scalar is now tightly coupled with Git, it does not make sense for
them to show different versions. Therefore, it shows the same output as
`git versions`. For backwards-compatibility with the .NET version,
`scalar version` prints to `stderr`, though (`git version` prints to
`stdout` instead).
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 39 +++++++++++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)
On 8/30/21 5:34 PM, Johannes Schindelin via GitGitGadget wrote:
tl;dr: This series contributes the Scalar command to the Git project. This
command provides an opinionated way to create and configure repositories
with a focus on very large repositories.
I want to give Johannes a big thanks for organizing this RFC. As you
can see from the authorship of the patches, this was an amazingly
collaborative effort, but Johannes led the way by creating a base that
the rest of us could work with, then finally he brought in all of the
gritty details to finish the effort.
Background
==========
...
The Scalar project
was created to make that separation, refine the key concepts, and then
extract those features into the new Scalar command.
When people have asked me how Scalar fits with the core Git client, I
point them to our "Philosophy of Scalar" document [1]. The most concise
summary of our goals since starting Scalar has been that Scalar aligns
with features already within Git that enable scale. I've said several
times that we are constantly making Scalar do less by making Git do more.
[1] https://github.com/microsoft/git/blob/HEAD/contrib/scalar/docs/philosophy.md
Here is an example: when our large, internal customer told us that they
required Linux support for Scalar, we looked at what it would take. We
could have done the necessary platform-specific things to convince .NET
Core to create a long-running process that launched Git maintenance tasks
at different intervals, creating a similar mechanism to the Windows and
macOS services that did those operations. But we also knew that the
existing system was stuck with architectural decisions from VFS for Git
that were not actually in service of how Scalar worked. Instead, we
decided to build background maintenance into Git itself and had our Linux
port of Scalar run "git maintenance start".
Once the Linux port was proven out with Git's background maintenance, we
realized that the window where a user actually interacts with Scalar instead
of Git is extremely narrow: users run "scalar clone" or "scalar register"
and otherwise only run Git commands. The Scalar process does not need to
exist outside of that. (There are some other helpers that can be used in
a pinch to diagnose and fix problems, but they are rarely used. These
commands, such as 'scalar diagnose' can be contributed separately.)
It became clear that for our own needs it would be easier to ship one
installer that included the microsoft/git fork and the Scalar CLI, and
it would be simple to rewrite the Scalar CLI with all of the Git helper
APIs. We organized the code in a way that we thought would be amenable
to an upstream contribution (by placing in contrib/ and using Git code
style).
The thing about these commands is that they are _opinionated_. We rely
on these opinions for important internal users, but we realize that they
are not necessarily optimal for all users. Hence, we did not think it
wise to push those opinions onto the 'git' executable. Having 'scalar'
continue to live as a separate executable made sense to us.
I believe that by contributing Scalar to the full community, that we
create opportunities for Git in the future. For one, users and Git
distributors can opt into compiling Scalar so it is more available
to users who are interested. Another hopeful idea is that maybe this
reinvigorates ideas of how to streamline Git clones for large repos
without users needing to learn each and every knob to twist to get
things working. Since the Scalar CLI is contributed in the full
license of the Git project, pieces of it can be adapted into Git
proper as needed.
I look forward to hearing your thoughts.
Thanks,
-Stolee
From: Eric Sunshine <hidden> Date: 2021-08-31 06:19:16
On Mon, Aug 30, 2021 at 5:35 PM Johannes Schindelin via GitGitGadget
[off-list ref] wrote:
quoted hunk
After a Scalar upgrade, it can come in really handy if there is an easy
way to reconfigure all Scalar enlistments. This new option offers this
functionality.
Signed-off-by: Johannes Schindelin <redacted>
---
@@ -121,6 +121,10 @@ After a Scalar upgrade, or when the configuration of a Scalar enlistment+With the `--all` option, all enlistments currently registered with Scalar+will be reconfigured. This option is meant to to be run every time Scalar+was upgraded.
From: Eric Sunshine <hidden> Date: 2021-08-31 06:24:50
On Mon, Aug 30, 2021 at 5:35 PM Johannes Schindelin via GitGitGadget
[off-list ref] wrote:
The .NET version of Scalar has a `version` command. This was necessary
because it was versioned independently of Git.
Since Scalar is now tightly coupled with Git, it does not make sense for
them to show different versions. Therefore, it shows the same output as
`git versions`. For backwards-compatibility with the .NET version,
s/versions/version/
`scalar version` prints to `stderr`, though (`git version` prints to
`stdout` instead).
Signed-off-by: Johannes Schindelin <redacted>
On Mon, Aug 30 2021, Derrick Stolee via GitGitGadget wrote:
[...]
+#ifndef WIN32
+ { "core.untrackedCache", "true" },
+#else
+ /*
+ * Unfortunately, Scalar's Functional Tests demonstrated
+ * that the untracked cache feature is unreliable on Windows
+ * (which is a bummer because that platform would benefit the
+ * most from it). For some reason, freshly created files seem
+ * not to update the directory's `lastModified` time
+ * immediately, but the untracked cache would need to rely on
+ * that.
+ *
+ * Therefore, with a sad heart, we disable this very useful
+ * feature on Windows.
+ */
+ { "core.untrackedCache", "false" },
+#endif
[...]
Ok, but why the need to set it to "false" explicitly? Does it need to be
so opinionated as to overwrite existing user-set config in these cases?
The commit message doesn't discuss these trace2 additions, these in
particular seem like they might be useful, but better done as as some
more general trace2 intergration in config.c, i.e. if the functions
being called here did the same logging on config set/get.
On Mon, Aug 30 2021, Johannes Schindelin via GitGitGadget wrote:
To test the Scalar command, create a test script in contrib/scalar/t
that is executed as `make -C contrib/scalar test`. Since Scalar has no
meaningful capabilities yet, the only test is rather simple. We will add
more tests in subsequent commits that introduce corresponding, new
functionality.
As a comment on 01..03/15: I'd really prefer if we stop using this
pattern of sub-Makefile, the dependencies are a pain to manage, and we
end up copy/pasting large sets of functionality.
That would mean just adding the build of this command to the top-level
Makefile behind some "CONTRIB_SCALAR" flag or whatever, but I find that
much cleaner than....
[...]
+../../bin-wrappers/scalar: ../../wrap-for-bin.sh Makefile
[...]
scalar.html: | scalar.1 # prevent them from trying to build `doc.dep` in parallel
...things like this, which refer to assets built by other Makefiles, and
need to plaster over the dependency issues...
quoted hunk
+++ b/contrib/scalar/t/Makefile
@@ -0,0 +1,78 @@+# Run scalar tests+#+# Copyright (c) 2005,2021 Junio C Hamano, Johannes Schindelin+#++-include ../../../config.mak.autogen+-include ../../../config.mak++SHELL_PATH?=$(SHELL)+PERL_PATH?=/usr/bin/perl+RM?=rm-f+PROVE?=prove+DEFAULT_TEST_TARGET?=test+TEST_LINT?=test-lint++ifdef TEST_OUTPUT_DIRECTORY+TEST_RESULTS_DIRECTORY=$(TEST_OUTPUT_DIRECTORY)/test-results+else+TEST_RESULTS_DIRECTORY=../../../t/test-results+endif++# Shell quote;+SHELL_PATH_SQ=$(subst','\'',$(SHELL_PATH))+PERL_PATH_SQ=$(subst','\'',$(PERL_PATH))+TEST_RESULTS_DIRECTORY_SQ=$(subst','\'',$(TEST_RESULTS_DIRECTORY))++T=$(sort$(wildcardt[0-9][0-9][0-9][0-9]-*.sh))++all:$(DEFAULT_TEST_TARGET)++test:$(TEST_LINT)+$(MAKE)aggregate-results-and-cleanup++prove:$(TEST_LINT)+@echo"*** prove ***";GIT_CONFIG=.git/config$(PROVE)--exec'$(SHELL_PATH_SQ)'$(GIT_PROVE_OPTS)$(T)::$(GIT_TEST_OPTS)+$(MAKE)clean-except-prove-cache++$(T):+@echo"*** $@ ***";GIT_CONFIG=.git/config'$(SHELL_PATH_SQ)'$@$(GIT_TEST_OPTS)++clean-except-prove-cache:+$(RM)-r'trash directory'.*'$(TEST_RESULTS_DIRECTORY_SQ)'+$(RM)-rvalgrind/bin++clean:clean-except-prove-cache+$(RM).prove++test-lint:test-lint-duplicatestest-lint-executabletest-lint-shell-syntax++test-lint-duplicates:+@dups=`echo$(T)|tr' ''\n'|sed's/-.*//'|sort|uniq-d`&&\+test-z"$$dups"||{\+echo>&2"duplicate test numbers:"$$dups;exit1;}++test-lint-executable:+@bad=`foriin$(T);dotest-x"$$i"||echo$$i;done`&&\+test-z"$$bad"||{\+echo>&2"non-executable tests:"$$bad;exit1;}++test-lint-shell-syntax:+@'$(PERL_PATH_SQ)'../../../t/check-non-portable-shell.pl$(T)++aggregate-results-and-cleanup:$(T)+$(MAKE)aggregate-results+$(MAKE)clean++aggregate-results:+forfin'$(TEST_RESULTS_DIRECTORY_SQ)'/t*-*.counts;do\+echo"$$f";\+done|'$(SHELL_PATH_SQ)'../../../t/aggregate-results.sh++valgrind:+$(MAKE)GIT_TEST_OPTS="$(GIT_TEST_OPTS) --valgrind"++test-results:+mkdir-ptest-results++.PHONY:$(T)aggregate-resultscleanvalgrind
...and this entire copy/pasting & adjusting of t/Makefile.
On Mon, Aug 30 2021, Johannes Schindelin via GitGitGadget wrote:
This implements Scalar's opinionated `clone` command: it tries to use a
partial clone and sets up a sparse checkout by default. In contrast to
`git clone`, `scalar clone` sets up the worktree in the `src/`
subdirectory, to encourage a separation between the source files and the
build output (which helps Git tremendously because it avoids untracked
files that have to be specifically ignored when refreshing the index).
Perhaps nobody else wondered this while reading this, but I thought this
might be some sparse/worktree magic where cloning into "foo" would have
"foo/.git", but the worktree was somehow magically mapped at foo/src/".
But no, it just takes your "scalar clone <url> foo" and translates it to
"foo/src", so you'll get a directory at "foo".
Note: We intentionally use a slightly wasteful `set_config()` function
(which does not reuse a single `strbuf`, for example, though performance
_really_ does not matter here) for convenience and readability.
FWIW I think the commit message could do without this, that part of the
code is obviously not performance sensitive at all. But maybe an
explicit note helps anyway...
You won't need the churn/boilerplate of adding "1" to everything here,
but can just change the initial patch to use designated initializers.
That along with a throwaway macro like:
#define SCALAR_CFG_TRUE(k) (.key = k, .value = "true")
#define SCALAR_CFG_FALSE(k) (.key = k, .value = "false")
Might (or might not) make this even easier to eyeball...
On Mon, Aug 30 2021, Johannes Schindelin via GitGitGadget wrote:
The `git` executable has these two very useful options:
-C <directory>:
switch to the specified directory before performing any actions
-c <key>=<value>:
temporarily configure this setting for the duration of the
specified scalar subcommand
With this commit, we teach the `scalar` executable the same trick.
[...]
+ while (argc > 1 && *argv[1] == '-') {
+ if (!strcmp(argv[1], "-C")) {
+ if (argc < 3)
+ die(_("-C requires a <directory>"));
+ if (chdir(argv[2]) < 0)
+ die_errno(_("could not change to '%s'"),
+ argv[2]);
+ argc -= 2;
+ argv += 2;
+ } else if (!strcmp(argv[1], "-c")) {
+ if (argc < 3)
+ die(_("-c requires a <key>=<value> argument"));
+ git_config_push_parameter(argv[2]);
+ argc -= 2;
+ argv += 2;
+ } else
+ break;
+ }
This along with my earlier comment about the Makefile copy/pasting makes
me wonder if an easier way to integrate this wouldn't be to refactor
git.c a bit to have it understand either "git" or "scalar", then instead
of "ls-tree" etc. as "git" the subcommands would become "built-ins".
Which would give us both "[git|scalar] [-c ...] <cmd>" for free, and
elimante the need for the inevetable future divergence of wanting -p,
-P, --exec-path etc. in both.
On 8/31/2021 4:11 AM, Ævar Arnfjörð Bjarmason wrote:
On Mon, Aug 30 2021, Derrick Stolee via GitGitGadget wrote:
quoted
[...]
+#ifndef WIN32
+ { "core.untrackedCache", "true" },
+#else
+ /*
+ * Unfortunately, Scalar's Functional Tests demonstrated
+ * that the untracked cache feature is unreliable on Windows
+ * (which is a bummer because that platform would benefit the
+ * most from it). For some reason, freshly created files seem
+ * not to update the directory's `lastModified` time
+ * immediately, but the untracked cache would need to rely on
+ * that.
+ *
+ * Therefore, with a sad heart, we disable this very useful
+ * feature on Windows.
+ */
+ { "core.untrackedCache", "false" },
+#endif
[...]
Ok, but why the need to set it to "false" explicitly? Does it need to be
so opinionated as to overwrite existing user-set config in these cases?
Users can overwrite this local config value, but this is placed to avoid
a global config value from applying specifically within Scalar-created
repos.
quoted
+ { "core.bare", "false" },
Shouldn't this be set by "git init" already?
This one is probably a bit _too_ defensive. It can be removed.
quoted
[...]
+ { "core.logAllRefUpdates", "true" },
An opinionated thing unrelated to performance?
It's an opinionated thing related to supporting monorepo users. It helps
us diagnose issues they have by recreating a sequence of events.
The commit message doesn't discuss these trace2 additions, these in
particular seem like they might be useful, but better done as as some
more general trace2 intergration in config.c, i.e. if the functions
being called here did the same logging on config set/get.
If we want to do such a tracing change within git_config_set*(), then
that would be an appropriate replacement. The biggest reason to include
them here is to trace that an existing value already exists, for the
case of running 'scalar reconfigure' during an upgrade. That part
doesn't make much sense to put into config.c.
Thanks,
-Stolee
On 8/31/2021 4:32 AM, Ævar Arnfjörð Bjarmason wrote:
On Mon, Aug 30 2021, Johannes Schindelin via GitGitGadget wrote:
quoted
The `git` executable has these two very useful options:
-C <directory>:
switch to the specified directory before performing any actions
-c <key>=<value>:
temporarily configure this setting for the duration of the
specified scalar subcommand
With this commit, we teach the `scalar` executable the same trick.
[...]
+ while (argc > 1 && *argv[1] == '-') {
+ if (!strcmp(argv[1], "-C")) {
+ if (argc < 3)
+ die(_("-C requires a <directory>"));
+ if (chdir(argv[2]) < 0)
+ die_errno(_("could not change to '%s'"),
+ argv[2]);
+ argc -= 2;
+ argv += 2;
+ } else if (!strcmp(argv[1], "-c")) {
+ if (argc < 3)
+ die(_("-c requires a <key>=<value> argument"));
+ git_config_push_parameter(argv[2]);
+ argc -= 2;
+ argv += 2;
+ } else
+ break;
+ }
This along with my earlier comment about the Makefile copy/pasting makes
me wonder if an easier way to integrate this wouldn't be to refactor
git.c a bit to have it understand either "git" or "scalar", then instead
of "ls-tree" etc. as "git" the subcommands would become "built-ins".
Which would give us both "[git|scalar] [-c ...] <cmd>" for free, and
elimante the need for the inevetable future divergence of wanting -p,
-P, --exec-path etc. in both.
Such a change would likely eliminate the ability to not include Scalar
when building the Git codebase, which we tried to avoid by keeping it
within contrib and have it be compiled via an opt-in flag.
If we want to talk about integrating Scalar into Git in a deeper way,
then that is an interesting discussion to have, but it lives at a much
higher level than Makefile details.
The questions we are really looking to answer in this RFC are:
1. Will the Git project accept Scalar into its codebase?
2. What is the best place for Scalar to live in the Git codebase?
We erred on the side of keeping Scalar as optional as possible. If
the community is more interested in a deeper integration, then that
could be an interesting direction.
In my opinion, I think the current tactic is safest. We could always
decide on a deeper integration later by moving the code around. It
seems harder to do the reverse.
Thanks,
-Stolee
On 8/31/2021 4:32 AM, Ævar Arnfjörð Bjarmason wrote:
quoted
On Mon, Aug 30 2021, Johannes Schindelin via GitGitGadget wrote:
quoted
The `git` executable has these two very useful options:
-C <directory>:
switch to the specified directory before performing any actions
-c <key>=<value>:
temporarily configure this setting for the duration of the
specified scalar subcommand
With this commit, we teach the `scalar` executable the same trick.
[...]
+ while (argc > 1 && *argv[1] == '-') {
+ if (!strcmp(argv[1], "-C")) {
+ if (argc < 3)
+ die(_("-C requires a <directory>"));
+ if (chdir(argv[2]) < 0)
+ die_errno(_("could not change to '%s'"),
+ argv[2]);
+ argc -= 2;
+ argv += 2;
+ } else if (!strcmp(argv[1], "-c")) {
+ if (argc < 3)
+ die(_("-c requires a <key>=<value> argument"));
+ git_config_push_parameter(argv[2]);
+ argc -= 2;
+ argv += 2;
+ } else
+ break;
+ }
This along with my earlier comment about the Makefile copy/pasting makes
me wonder if an easier way to integrate this wouldn't be to refactor
git.c a bit to have it understand either "git" or "scalar", then instead
of "ls-tree" etc. as "git" the subcommands would become "built-ins".
Which would give us both "[git|scalar] [-c ...] <cmd>" for free, and
elimante the need for the inevetable future divergence of wanting -p,
-P, --exec-path etc. in both.
Such a change would likely eliminate the ability to not include Scalar
when building the Git codebase, which we tried to avoid by keeping it
within contrib and have it be compiled via an opt-in flag.
I mean to still have it behind a flag, but to handle it similar to how
we handle NO_CURL, EXCLUDED_PROGRAMS and the like, i.e. not requiring
parallel maintenance of copy/pasted Makefile logic in contrib/.
If we want to talk about integrating Scalar into Git in a deeper way,
then that is an interesting discussion to have, but it lives at a much
higher level than Makefile details.
To be clear I'm proposing no change at all in term of what happens when
you run "make install", just commenting on the implementation details of
how we arrange for things to be built and configured before that step.
I realize that this is following some prior art of
e.g. contrib/subtree/Makefile, but IMNSHO that approach is a historical
mistake we should be backing out of. There was some recent discussion of
this here:
https://lore.kernel.org/git/87pmz4ig4o.fsf@evledraar.gmail.com/
E.g. now we have some painful management of the depencency graph between
/Makefile and Documentation/Makefile requiring fixes like 56550ea7180
(Makefile: add missing dependencies of 'config-list.h', 2021-04-08),
adding yet another Makefile into the mix which (to take one example)
depends on doc.dep, which in turn depends on ...; It's all a bunch of
needless complexity we can avoid.
The questions we are really looking to answer in this RFC are:
1. Will the Git project accept Scalar into its codebase?
2. What is the best place for Scalar to live in the Git codebase?
We erred on the side of keeping Scalar as optional as possible. If
the community is more interested in a deeper integration, then that
could be an interesting direction.
Indeed, to be clear I realize I'm entirely punting on the real questions
you're interested in. I just gave this an initial cursory skimming for
now, I have not formed an informed opinion on your #1, but just a little
bit of #2.
My initial reaction to #1 without having looked into it deeply is some
combination of "sure, why not?", and that the people/group contributing
major scalability work to git.git should be given the benefit of the
doubt. Maybe we won't keep "scalar" long-term, or change its UI etc.,
all of that can be handled in some carefully worded documentation
somewhere.
Of course all these suggestions I'm making about Makefile arrangement
are rather pointless if there isn't consensus to get past the hurdle of
your #1.
In my opinion, I think the current tactic is safest. We could always
decide on a deeper integration later by moving the code around. It
seems harder to do the reverse.
I think "deeper integration" is the reverse of what you think it is.
I.e. if I'm patching or maintaining part of the Makefile logic to it's
deeper (or perhaps "gnarlier" is the righ word?) integration to need to
duplicate that work in two places, or always take into account that some
not-built-by-default-but-quite-common command's *.txt docs and *.sh
tests live in some unusual place for the purposes of CI, lint, tooling
etc.
In other words, it's a question of how much net complexity is being
added to the (build) system. That complexity doesn't automatically
reduce just because some files live in another directory, sometimes
that's an increase in complexity.
Whereas just conditionally adding it to some list in the top-level
Makefile (or Documentation/Makefile) is relatively maintenance-free, and
to our users / packagers the result should be the same or near enough.
It won't matter to them if building the optional thing is another "make"
command or just a flag to the existing "make" command.
From: Eric Sunshine <hidden> Date: 2021-08-31 16:47:21
On Tue, Aug 31, 2021 at 8:04 AM Ævar Arnfjörð Bjarmason
[off-list ref] wrote:
On Mon, Aug 30 2021, Johannes Schindelin via GitGitGadget wrote:
quoted
Note: We intentionally use a slightly wasteful `set_config()` function
(which does not reuse a single `strbuf`, for example, though performance
_really_ does not matter here) for convenience and readability.
FWIW I think the commit message could do without this, that part of the
code is obviously not performance sensitive at all. But maybe an
explicit note helps anyway...
FWIW, I also found this distracting; it takes the reader's attention
away from more important aspects of the patch. (But it alone is not
worth a re-roll; it was just a minor hiccup.)
On Mon, Aug 30, 2021 at 5:52 PM Derrick Stolee [off-list ref] wrote:
On 8/30/21 5:34 PM, Johannes Schindelin via GitGitGadget wrote:
quoted
tl;dr: This series contributes the Scalar command to the Git project. This
command provides an opinionated way to create and configure repositories
with a focus on very large repositories.
I want to give Johannes a big thanks for organizing this RFC. As you
can see from the authorship of the patches, this was an amazingly
collaborative effort, but Johannes led the way by creating a base that
the rest of us could work with, then finally he brought in all of the
gritty details to finish the effort.
quoted
Background
==========
...
quoted
The Scalar project
was created to make that separation, refine the key concepts, and then
extract those features into the new Scalar command.
When people have asked me how Scalar fits with the core Git client, I
point them to our "Philosophy of Scalar" document [1]. The most concise
summary of our goals since starting Scalar has been that Scalar aligns
with features already within Git that enable scale. I've said several
times that we are constantly making Scalar do less by making Git do more.
[1] https://github.com/microsoft/git/blob/HEAD/contrib/scalar/docs/philosophy.md
Here is an example: when our large, internal customer told us that they
required Linux support for Scalar, we looked at what it would take. We
could have done the necessary platform-specific things to convince .NET
Core to create a long-running process that launched Git maintenance tasks
at different intervals, creating a similar mechanism to the Windows and
macOS services that did those operations. But we also knew that the
existing system was stuck with architectural decisions from VFS for Git
that were not actually in service of how Scalar worked. Instead, we
decided to build background maintenance into Git itself and had our Linux
port of Scalar run "git maintenance start".
Once the Linux port was proven out with Git's background maintenance, we
realized that the window where a user actually interacts with Scalar instead
of Git is extremely narrow: users run "scalar clone" or "scalar register"
and otherwise only run Git commands. The Scalar process does not need to
exist outside of that. (There are some other helpers that can be used in
a pinch to diagnose and fix problems, but they are rarely used. These
commands, such as 'scalar diagnose' can be contributed separately.)
It became clear that for our own needs it would be easier to ship one
installer that included the microsoft/git fork and the Scalar CLI, and
it would be simple to rewrite the Scalar CLI with all of the Git helper
APIs. We organized the code in a way that we thought would be amenable
to an upstream contribution (by placing in contrib/ and using Git code
style).
The thing about these commands is that they are _opinionated_. We rely
on these opinions for important internal users, but we realize that they
are not necessarily optimal for all users. Hence, we did not think it
wise to push those opinions onto the 'git' executable. Having 'scalar'
continue to live as a separate executable made sense to us.
I believe that by contributing Scalar to the full community, that we
create opportunities for Git in the future. For one, users and Git
distributors can opt into compiling Scalar so it is more available
to users who are interested. Another hopeful idea is that maybe this
reinvigorates ideas of how to streamline Git clones for large repos
without users needing to learn each and every knob to twist to get
things working. Since the Scalar CLI is contributed in the full
license of the Git project, pieces of it can be adapted into Git
proper as needed.
I look forward to hearing your thoughts.
Thanks,
-Stolee
Looks like exciting stuff, you two. I'm behind on review as it is; I
still need to get back to Stolee's sparse-index add/rm/mv series, but
I'll try to circle back and take a look.
From: Johannes Schindelin <hidden> Date: 2021-09-03 15:21:18
Hi Eric,
On Tue, 31 Aug 2021, Eric Sunshine wrote:
On Tue, Aug 31, 2021 at 8:04 AM Ævar Arnfjörð Bjarmason
[off-list ref] wrote:
quoted
On Mon, Aug 30 2021, Johannes Schindelin via GitGitGadget wrote:
quoted
Note: We intentionally use a slightly wasteful `set_config()` function
(which does not reuse a single `strbuf`, for example, though performance
_really_ does not matter here) for convenience and readability.
FWIW I think the commit message could do without this, that part of the
code is obviously not performance sensitive at all. But maybe an
explicit note helps anyway...
FWIW, I also found this distracting; it takes the reader's attention
away from more important aspects of the patch. (But it alone is not
worth a re-roll; it was just a minor hiccup.)
Since I reworked the remote default branch parsing anyway, I removed this
paragraph from the commit message.
Ciao,
Dscho
From: Johannes Schindelin <hidden> Date: 2021-09-03 15:23:22
Hi Eric,
On Tue, 31 Aug 2021, Eric Sunshine wrote:
On Mon, Aug 30, 2021 at 5:35 PM Johannes Schindelin via GitGitGadget
[off-list ref] wrote:
quoted
After a Scalar upgrade, it can come in really handy if there is an easy
way to reconfigure all Scalar enlistments. This new option offers this
functionality.
Signed-off-by: Johannes Schindelin <redacted>
---
@@ -121,6 +121,10 @@ After a Scalar upgrade, or when the configuration of a Scalar enlistment+With the `--all` option, all enlistments currently registered with Scalar+will be reconfigured. This option is meant to to be run every time Scalar+was upgraded.
s/was/is/
I wanted to convey a temporal order, so I changed it to "every time after
Scalar is upgraded". Okay?
Ciao,
Dscho
From: Johannes Schindelin <hidden> Date: 2021-09-03 15:24:28
Hi Eric,
On Tue, 31 Aug 2021, Eric Sunshine wrote:
On Mon, Aug 30, 2021 at 5:35 PM Johannes Schindelin via GitGitGadget
[off-list ref] wrote:
quoted
The .NET version of Scalar has a `version` command. This was necessary
because it was versioned independently of Git.
Since Scalar is now tightly coupled with Git, it does not make sense for
them to show different versions. Therefore, it shows the same output as
`git versions`. For backwards-compatibility with the .NET version,
s/versions/version/
Thank you!
Dscho
quoted
`scalar version` prints to `stderr`, though (`git version` prints to
`stdout` instead).
Signed-off-by: Johannes Schindelin <redacted>
From: Johannes Schindelin <hidden> Date: 2021-09-03 15:50:11
Hi Ævar,
On Tue, 31 Aug 2021, Ævar Arnfjörð Bjarmason wrote:
On Mon, Aug 30 2021, Derrick Stolee via GitGitGadget wrote:
quoted
+ const char *usagestr[] = { NULL, NULL };
Missing usage strings?
This command will show a generated usage, i.e. a non-static string. It
therefore cannot be specified here already. See the `strbuf_*()` calls
populating `buf` and the `usagestr[0] = buf.buf;` assignment.
quoted
+ if (argc == 0)
Style nit (per style guide): s/argc == 0/!argc/g.
It is true that we often do this, but in this instance it would be
misleading: `argc` is a counter, not a Boolean.
quoted
+ if (!strcmp("all", argv[0]))
+ i = -1;
Style nit (per style guide): missing braces here.
The style guide specifically allows my preference to leave single-line
blocks without curlies.
Ciao,
Johannes
You won't need the churn/boilerplate of adding "1" to everything here,
but can just change the initial patch to use designated initializers.
That along with a throwaway macro like:
#define SCALAR_CFG_TRUE(k) (.key = k, .value = "true")
#define SCALAR_CFG_FALSE(k) (.key = k, .value = "false")
Might (or might not) make this even easier to eyeball...
To me, it makes things less readable. There is an entire section with the
header `/* Optional */` below, and I want this list to stay as readable as
it is now.
Ciao,
Dscho
From: Eric Sunshine <hidden> Date: 2021-09-03 17:03:03
On Fri, Sep 3, 2021 at 11:23 AM Johannes Schindelin
[off-list ref] wrote:
On Tue, 31 Aug 2021, Eric Sunshine wrote:
quoted
On Mon, Aug 30, 2021 at 5:35 PM Johannes Schindelin via GitGitGadget
quoted
+With the `--all` option, all enlistments currently registered with Scalar
+will be reconfigured. This option is meant to to be run every time Scalar
+was upgraded.
s/was/is/
I wanted to convey a temporal order, so I changed it to "every time after
Scalar is upgraded". Okay?
I think I understood the intent of the original, but it causes a
grammatical hiccup. Your revised version can work, although I might
write it this way:
This option is meant to be run each time Scalar is upgraded.
However, perhaps that is too ambiguous and some users may think that
the process of upgrading Scalar will automatically run this command,
and you'd like to make it clear that it is the user's responsibility.
So, perhaps:
Use this option after each Scalar upgrade.
or something.
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-09-03 17:54:54
tl;dr: This series contributes the Scalar command to the Git project. This
command provides an opinionated way to create and configure repositories
with a focus on very large repositories.
Background
==========
Years ago, Microsoft wanted to move the source code of the Windows operating
system to Git. The challenge there was to prove that Git could scale to
massive monorepos. The VFS for Git (formerly GVFS) project was born to take
up that challenge.
The final solution included a virtual filesystem (with both user-mode and
kernel components) and a customized fork of Git for Windows. This solution
contained several key concepts, such as only populating a portion of the
working directory, demand-fetching blobs, and performing periodic repo
maintenance in the background. However, the required kernel drivers made it
difficult to port the solution to other platforms.
But it was realized that many of these key concepts were independent of the
actual VFS and its projection of the working directory. The Scalar project
was created to make that separation, refine the key concepts, and then
extract those features into the new Scalar command.
The present
===========
The Scalar project provides a completely functional non-virtual experience
for monorepos. But why stop there. The Scalar project was designed to be a
self-destructing vehicle to allow those key concepts to be moved into core
Git itself for the benefit of all. For example, partial clone,
sparse-checkout, and background maintenance have already been upstreamed and
removed from Scalar proper. This patch series provides a C-based
implementation of the final remaining portions of the Scalar command. This
will make it easier for users to experiment with the Scalar command. It will
also make it substantially easier to experiment with moving functionality
from Scalar into core Git, while maintaining backwards-compatibility for
existing Scalar users.
The C-based Scalar has been shipped to Scalar users, and can be tested by
any interested reader:
https://github.com/microsoft/git/releases/tag/v2.33.0.vfs.0.0 (it offers a
Git for Windows installer, a macOS package and an Ubuntu package).
Opportunities
=============
Apart from providing the Scalar command, this contribution is intended to
serve as a basis for further mailing list discussions on moving (some of)
these key concepts into the main Git commands.
For example, we previously discussed the idea of a "git big-clone" that does
much of what "scalar clone" is doing. This patch series is a step to make
such functionality exist in the Git code base while we simmer on what such a
"git big-clone" command-line interface would look like.
This is one of many possible ways to do this. Creating a 'git big-clone'
could lock Git into backwards compatibility concerns so it is necessary to
approach such an endeavor with caution. As a discussion starter, the scalar
clone <url> command does roughly this:
1. git clone --sparse --filter=blob:none /src
2. git -C /src sparse-checkout init --cone
3. git -C /src config (many times)
4. git -C /src maintenance start
It is my hope inspire discussions about what parts of Scalar could go into
core Git, and where, and in which form. While we wish to maintain
backwards-compatibility of Scalar's command-line interface (because it is
already in use), by having the Scalar code in the same code base as Git's,
it will be much easier to move functionality without having to maintain
loose version coupling between independently-versioned Scalar and Git. The
tight version-coupling, along with having access to libgit.a also allows the
C-based implementation of Scalar to be much smaller than the original .NET
version.
For example, we might choose in the future to implement, say, git clone
--scale=partial,cone to initialize a partial clone with a cone-sparse
checkout, that would not only be totally doable, and not only would we
already have precedent and data to prove that this actually makes engineers
happy who have to work on ginormous repositories, but we could then also
implement it by moving parts of contrib/scalar/ to builtin/ (where
contrib/scalar/ would then call the built-ins accordingly rather than
hard-coding the defaults itself).
We now also have the opportunity to discuss the merits of Scalar's clone
caching, which is not actually part of this patch series because it is a bit
coupled with the GVFS parts of microsoft/git for the moment, where clones
automatically get registered with a populated alternate repository that is
identified by the URL, meaning: subsequent clones of the same repository are
vastly faster than the first one because they do not actually download the
already-received objects again, they access the cache instead.
Another thing that I could imagine to be discussed at length is the
distinction between enlistment and worktree (where the latter is the actual
Git worktree and usually lives in the src/ subdirectory of the former). This
encourages untracked and ignored files to be placed outside the worktree,
making Git's job much easier. This idea, too, might find its way in one way
or another into Git proper.
These are just a few concepts in Scalar that do not yet have equivalents in
Git. By putting this initial implementation into contrib/, we create a
foundation for future discussions of these concepts.
We plan on updating the recommended config settings in scalar register as
new Git features are available (such as builtin FSMonitor and sparse-index,
when ready). To facilitate upgrading existing Scalar enlistments, their
paths are automatically added to the [scalar] section of the global Git
config, and the scalar reconfigure --all command will process all of them.
Epilogue
========
Now, to address some questions that I imagine every reader has who made it
this far:
* Why not put the Scalar functionality directly into a built-in? Creating a
Git builtin requires scrutiny over every aspect of the feature, which is
difficult to do while also maintaining the command-line interface
contract and expected behavior of the Scalar command (there are existing
users, after all). By having the Scalar command in contrib/, we present a
simple option for users to have these features in the short term while
the Git contributor community decides which bits to absorb into Git
built-ins.
* Why implement the Scalar command in the Git codebase? We ported Scalar to
the microsoft/git fork for several reasons. First, we realized it was
possible now that the core features exist inside Git itself. Second,
compiling Scalar directly within a version of Git allows us to remove a
version compatibility check from each config option that might or might
not apply based on the installed Git version. Finally, this new location
has greatly simplified our release process and the installation process
for users. We now have ways to install Scalar with microsoft/git via
winget, brew, and apt-get. This has been the case since we shipped
v2.32.0 to our users, read: this setup has served us well already.
* Why contribute Scalar to the Git project? We are biased, of course, yet
we do have evidence that the Scalar command is a helpful tool that offers
an simple way to handle huge repositories with ease. By contributing it
to the core Git project, we are able to share it with more users,
especially some users who do not want to install the microsoft/git fork.
We intend to include Scalar as a component in git-for-windows/git, but
are contributing it here first. Further, we think there is benefit to the
Git developer community as this presents an example of how to set certain
defaults that work for large repositories.
* Does this integrate with the built-in FSMonitor yet? No, not yet. I do
have a couple of add-on patch series lined up, one of them being the
integration with the built-in FSMonitor, which obviously has to wait
until the FSMonitor patch series advances further.
Changes since v1:
* A couple typos were fixed
* The code parsing the output of ls-remote was made more readable
* The indentation used in scalar.txt now consistently uses tabs
* We no longer hard-code core.bare = false when registering with Scalar
Derrick Stolee (4):
scalar: 'register' sets recommended config and starts maintenance
scalar: 'unregister' stops background maintenance
scalar: implement 'scalar list'
scalar: implement the `run` command
Johannes Schindelin (10):
scalar: create a rudimentary executable
scalar: start documenting the command
scalar: create test infrastructure
scalar: let 'unregister' handle a deleted enlistment directory
gracefully
scalar: implement the `clone` subcommand
scalar: teach 'clone' to support the --single-branch option
scalar: allow reconfiguring an existing enlistment
scalar: teach 'reconfigure' to optionally handle all registered
enlistments
scalar: implement the `version` command
scalar: accept -C and -c options before the subcommand
Matthew John Cheetham (1):
scalar: implement the `delete` command
Makefile | 8 +
contrib/scalar/.gitignore | 5 +
contrib/scalar/Makefile | 57 +++
contrib/scalar/scalar.c | 844 +++++++++++++++++++++++++++++++
contrib/scalar/scalar.txt | 156 ++++++
contrib/scalar/t/Makefile | 78 +++
contrib/scalar/t/t9099-scalar.sh | 88 ++++
7 files changed, 1236 insertions(+)
create mode 100644 contrib/scalar/.gitignore
create mode 100644 contrib/scalar/Makefile
create mode 100644 contrib/scalar/scalar.c
create mode 100644 contrib/scalar/scalar.txt
create mode 100644 contrib/scalar/t/Makefile
create mode 100755 contrib/scalar/t/t9099-scalar.sh
base-commit: ebf3c04b262aa27fbb97f8a0156c2347fecafafb
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1005%2Fdscho%2Fscalar-the-beginning-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1005/dscho/scalar-the-beginning-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/1005
Range-diff vs v1:
1: b8c7d3f8450 = 1: b8c7d3f8450 scalar: create a rudimentary executable
2: 4f886575dcf = 2: 4f886575dcf scalar: start documenting the command
3: bcfde9bc765 = 3: bcfde9bc765 scalar: create test infrastructure
4: 3786f4c597f ! 4: ee3e26a0c4e scalar: 'register' sets recommended config and starts maintenance
@@ contrib/scalar/scalar.c
+ */
+ { "core.untrackedCache", "false" },
+#endif
-+ { "core.bare", "false" },
+ { "core.logAllRefUpdates", "true" },
+ { "credential.https://dev.azure.com.useHttpPath", "true" },
+ { "credential.validate", "false" }, /* GCM4W-only */
@@ contrib/scalar/scalar.txt: will be identical to the worktree.
+~~~~~~~~
+
+register [<enlistment>]::
-+ Adds the enlistment's repository to the list of registered repositories
-+ and starts background maintenance. If `<enlistment>` is not provided,
-+ then the enlistment associated with the current working directory is
-+ registered.
++ Adds the enlistment's repository to the list of registered repositories
++ and starts background maintenance. If `<enlistment>` is not provided,
++ then the enlistment associated with the current working directory is
++ registered.
+++
++Note: when this subcommand is called in a worktree that is called `src/`, its
++parent directory is considered to be the Scalar enlistment. If the worktree is
++_not_ called `src/`, it itself will be considered to be the Scalar enlistment.
+
SEE ALSO
--------
5: 2a6ac170e6b ! 5: 6142f75875b scalar: 'unregister' stops background maintenance
@@ contrib/scalar/scalar.txt: SYNOPSIS
DESCRIPTION
-----------
-@@ contrib/scalar/scalar.txt: register [<enlistment>]::
- then the enlistment associated with the current working directory is
- registered.
+@@ contrib/scalar/scalar.txt: Note: when this subcommand is called in a worktree that is called `src/`, its
+ parent directory is considered to be the Scalar enlistment. If the worktree is
+ _not_ called `src/`, it itself will be considered to be the Scalar enlistment.
+Unregister
+~~~~~~~~~~
+
+unregister [<enlistment>]::
-+ Remove the specified repository from the list of repositories
-+ registered with Scalar and stop the scheduled background maintenance.
++ Remove the specified repository from the list of repositories
++ registered with Scalar and stop the scheduled background maintenance.
+
SEE ALSO
--------
6: 087fc9be194 = 6: 82dd253154f scalar: let 'unregister' handle a deleted enlistment directory gracefully
7: c272ff4069d ! 7: fb7c931ddb3 scalar: implement 'scalar list'
@@ contrib/scalar/scalar.txt: an existing Git worktree with Scalar whose name is no
+~~~~
+
+list::
-+ To see which repositories are currently registered by the service, run
-+ `scalar list`. This subcommand does not need to be run inside a Scalar
-+ enlistment.
++ To see which repositories are currently registered by the service, run
++ `scalar list`. This subcommand does not need to be run inside a Scalar
++ enlistment.
+
Register
~~~~~~~~
8: 2cbf0b61113 ! 8: f3223c10788 scalar: implement the `clone` subcommand
@@ Commit message
experience and experiments of the Microsoft Windows and the Microsoft
Office development teams.
- Note: We intentionally use a slightly wasteful `set_config()` function
- (which does not reuse a single `strbuf`, for example, though performance
- _really_ does not matter here) for convenience and readability.
-
- Also note: since the `scalar clone` command is by far the most commonly
+ Note: since the `scalar clone` command is by far the most commonly
called `scalar` subcommand, we document it at the top of the manual
page.
@@ contrib/scalar/scalar.c: static int unregister_dir(void)
+
+ cp.git_cmd = 1;
+ strvec_pushl(&cp.args, "ls-remote", "--symref", url, "HEAD", NULL);
-+ strbuf_addstr(&out, "-\n");
+ if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) {
-+ char *ref = out.buf;
-+
-+ while ((ref = strstr(ref + 1, "\nref: "))) {
-+ const char *p;
-+ char *head, *branch;
++ const char *line = out.buf;
+
-+ ref += strlen("\nref: ");
-+ head = strstr(ref, "\tHEAD");
++ while (*line) {
++ const char *eol = strchrnul(line, '\n'), *p;
++ size_t len = eol - line;
++ char *branch;
+
-+ if (!head || memchr(ref, '\n', head - ref))
++ if (!skip_prefix(line, "ref: ", &p) ||
++ !strip_suffix_mem(line, &len, "\tHEAD")) {
++ line = eol + (*eol == '\n');
+ continue;
++ }
+
-+ if (skip_prefix(ref, "refs/heads/", &p)) {
-+ branch = xstrndup(p, head - p);
++ eol = line + len;
++ if (skip_prefix(p, "refs/heads/", &p)) {
++ branch = xstrndup(p, eol - p);
+ strbuf_release(&out);
+ return branch;
+ }
+
+ error(_("remote HEAD is not a branch: '%.*s'"),
-+ (int)(head - ref), ref);
++ (int)(eol - p), p);
+ strbuf_release(&out);
+ return NULL;
+ }
@@ contrib/scalar/scalar.txt: an existing Git worktree with Scalar whose name is no
+~~~~~
+
+clone [<options>] <url> [<enlistment>]::
-+ Clones the specified repository, similar to linkgit:git-clone[1]. By
-+ default, only commit and tree objects are cloned. Once finished, the
-+ worktree is located at `<enlistment>/src`.
++ Clones the specified repository, similar to linkgit:git-clone[1]. By
++ default, only commit and tree objects are cloned. Once finished, the
++ worktree is located at `<enlistment>/src`.
++
+The sparse-checkout feature is enabled (except when run with `--full-clone`)
+and the only files present are those in the top-level directory. Use
@@ contrib/scalar/scalar.txt: an existing Git worktree with Scalar whose name is no
+
+-b <name>::
+--branch <name>::
-+ Instead of checking out the branch pointed to by the cloned repository's
-+ HEAD, check out the `<name>` branch instead.
++ Instead of checking out the branch pointed to by the cloned
++ repository's HEAD, check out the `<name>` branch instead.
+
+--[no-]full-clone::
-+ A sparse-checkout is initialized by default. This behavior can be turned
-+ off via `--full-clone`.
++ A sparse-checkout is initialized by default. This behavior can be
++ turned off via `--full-clone`.
+
List
~~~~
list::
- To see which repositories are currently registered by the service, run
-- `scalar list`. This subcommand does not need to be run inside a Scalar
-- enlistment.
-+ `scalar list`. This subcommand, like `clone`, does not need to be run
-+ inside a Scalar enlistment.
+ To see which repositories are currently registered by the service, run
+- `scalar list`. This subcommand does not need to be run inside a Scalar
+- enlistment.
++ `scalar list`. This subcommand, like `clone`, does not need to be run
++ inside a Scalar enlistment.
Register
~~~~~~~~
9: 9af1c37c2ea ! 9: b3c4b3dccc6 scalar: teach 'clone' to support the --single-branch option
@@ contrib/scalar/scalar.txt: scalar - an opinionated repository management tool
scalar register [<enlistment>]
scalar unregister [<enlistment>]
@@ contrib/scalar/scalar.txt: subdirectories outside your sparse-checkout by using `git ls-tree HEAD`.
- Instead of checking out the branch pointed to by the cloned repository's
- HEAD, check out the `<name>` branch instead.
+ Instead of checking out the branch pointed to by the cloned
+ repository's HEAD, check out the `<name>` branch instead.
+--[no-]single-branch::
-+ Clone only the history leading to the tip of a single branch,
-+ either specified by the `--branch` option or the primary
-+ branch remote's `HEAD` points at.
++ Clone only the history leading to the tip of a single branch, either
++ specified by the `--branch` option or the primary branch remote's
++ `HEAD` points at.
++
+Further fetches into the resulting repository will only update the
+remote-tracking branch for the branch this option was used for the initial
@@ contrib/scalar/scalar.txt: subdirectories outside your sparse-checkout by using
+`--single-branch` clone was made, no remote-tracking branch is created.
+
--[no-]full-clone::
- A sparse-checkout is initialized by default. This behavior can be turned
- off via `--full-clone`.
+ A sparse-checkout is initialized by default. This behavior can be
+ turned off via `--full-clone`.
## contrib/scalar/t/t9099-scalar.sh ##
@@ contrib/scalar/t/t9099-scalar.sh: test_expect_success 'set up repository to clone' '
10: c3f16bccd02 ! 10: b7fc2dc29c8 scalar: implement the `run` command
@@ contrib/scalar/scalar.txt: scalar clone [--single-branch] [--branch <main-branch
DESCRIPTION
-----------
@@ contrib/scalar/scalar.txt: unregister [<enlistment>]::
- Remove the specified repository from the list of repositories
- registered with Scalar and stop the scheduled background maintenance.
+ Remove the specified repository from the list of repositories
+ registered with Scalar and stop the scheduled background maintenance.
+Run
+~~~
+
+scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]::
-+ Run the given maintenance task (or all tasks, if `all` was specified).
-+ Except for `all` and `config`, this subcommand simply hands off to
-+ linkgit:git-maintenance[1] (mapping `fetch` to `prefetch` and
-+ `pack-files` to `incremental-repack`).
++ Run the given maintenance task (or all tasks, if `all` was specified).
++ Except for `all` and `config`, this subcommand simply hands off to
++ linkgit:git-maintenance[1] (mapping `fetch` to `prefetch` and
++ `pack-files` to `incremental-repack`).
++
+These tasks are run automatically as part of the scheduled maintenance,
+as soon as the repository is registered with Scalar. It should therefore
11: 13056f02018 ! 11: 9a834c23d08 scalar: allow reconfiguring an existing enlistment
@@ contrib/scalar/scalar.c: static int set_recommended_config(void)
- { "core.untrackedCache", "false" },
+ { "core.untrackedCache", "false", 1 },
#endif
-- { "core.bare", "false" },
- { "core.logAllRefUpdates", "true" },
- { "credential.https://dev.azure.com.useHttpPath", "true" },
- { "credential.validate", "false" }, /* GCM4W-only */
@@ contrib/scalar/scalar.c: static int set_recommended_config(void)
- { "feature.experimental", "false" },
- { "fetch.unpackLimit", "1" },
- { "fetch.writeCommitGraph", "false" },
-+ { "core.bare", "false", 1 },
+ { "core.logAllRefUpdates", "true", 1 },
+ { "credential.https://dev.azure.com.useHttpPath", "true", 1 },
+ { "credential.validate", "false", 1 }, /* GCM4W-only */
12: 732a28c22fc ! 12: 79e9f5d203a scalar: teach 'reconfigure' to optionally handle all registered enlistments
@@ contrib/scalar/scalar.txt: After a Scalar upgrade, or when the configuration of
reconfigure the enlistment.
+With the `--all` option, all enlistments currently registered with Scalar
-+will be reconfigured. This option is meant to to be run every time Scalar
-+was upgraded.
++will be reconfigured. This option is meant to to be run every time after
++Scalar is upgraded.
+
SEE ALSO
--------
13: 13afbd68812 ! 13: 94a21982652 scalar: implement the `delete` command
@@ contrib/scalar/scalar.txt: scalar register [<enlistment>]
DESCRIPTION
-----------
@@ contrib/scalar/scalar.txt: With the `--all` option, all enlistments currently registered with Scalar
- will be reconfigured. This option is meant to to be run every time Scalar
- was upgraded.
+ will be reconfigured. This option is meant to to be run every time after
+ Scalar is upgraded.
+Delete
+~~~~~~
+
+delete <enlistment>::
-+ This subcommand lets you delete an existing Scalar enlistment from your
-+ local file system, unregistering the repository.
++ This subcommand lets you delete an existing Scalar enlistment from your
++ local file system, unregistering the repository.
+
SEE ALSO
--------
14: 73d08c0c894 ! 14: 707d8e19683 scalar: implement the `version` command
@@ Commit message
Since Scalar is now tightly coupled with Git, it does not make sense for
them to show different versions. Therefore, it shows the same output as
- `git versions`. For backwards-compatibility with the .NET version,
+ `git version`. For backwards-compatibility with the .NET version,
`scalar version` prints to `stderr`, though (`git version` prints to
`stdout` instead).
15: 6455b18f1b6 ! 15: 26e23b5c5e5 scalar: accept -C and -c options before the subcommand
@@ contrib/scalar/scalar.txt: The `scalar` command implements various subcommands,
+The following options can be specified _before_ the subcommand:
+
+-C <directory>::
-+ Before running the subcommand, change the working directory. This
-+ option imitates the same option of linkgit:git[1].
++ Before running the subcommand, change the working directory. This
++ option imitates the same option of linkgit:git[1].
+
+-c <key>=<value>::
-+ For the duration of running the specified subcommand, configure this
-+ setting. This option imitates the same option of linkgit:git[1].
++ For the duration of running the specified subcommand, configure this
++ setting. This option imitates the same option of linkgit:git[1].
+
COMMANDS
--------
--
gitgitgadget
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-09-03 17:54:56
From: Johannes Schindelin <redacted>
The idea of Scalar (https://github.com/microsoft/scalar), and before
that, of VFS for Git, has always been to prove that Git _can_ scale, and
to upstream whatever strategies have been demonstrated to help.
With this patch, we start the journey from that C# project to move what
is left to Git's own `contrib/` directory, reimplementing it in pure C,
with the intention to facilitate integrating the functionality into core
Git all while maintaining backwards-compatibility for existing Scalar
users (which will be much easier when both live in the same worktree).
It was always to plan to contribute all of the proven strategies back to
core Git.
For example, while the virtual filesystem provided by VFS for Git helped
the team developing the Windows operating system to move onto Git, while
trying to upstream it we realized that it cannot be done: getting the
virtual filesystem to work (which we only managed to implement fully on
Windows, but not on, say, macOS or Linux), and the required server-side
support for the GVFS protocol, made this not quite feasible.
The Scalar project learned from that and tackled the problem with
different tactics: instead of pretending to Git that the working
directory is fully populated, it _specifically_ teaches Git about
partial clone (which is based on VFS for Git's cache server), about
sparse checkout (which VFS for Git tried to do transparently, in the
file system layer), and regularly runs maintenance tasks to keep the
repository in a healthy state.
With partial clone, sparse checkout and `git maintenance` having been
upstreamed, there is little left that `scalar.exe` does that which
`git.exe` cannot do. One such thing is that `scalar clone <url>` will
automatically set up a partial, sparse clone, and configure
known-helpful settings from the start.
So let's bring this convenience into Git's tree.
The idea here is that you can (optionally) build Scalar via
make -C contrib/scalar/Makefile
This will build the `scalar` executable and put it into the
contrib/scalar/ subdirectory.
The slightly awkward addition of the `contrib/scalar/*` bits to the
top-level `Makefile` are actually really required: we want to link to
`libgit.a`, which means that we will need to use the very same `CFLAGS`
and `LDFLAGS` as the rest of Git.
An early development version of this patch tried to replicate all the
conditional code in `contrib/scalar/Makefile` (e.g. `NO_POLL`) just like
`contrib/svn-fe/Makefile` used to do before it was retired. It turned
out to be quite the whack-a-mole game: the SHA-1-related flags, the
flags enabling/disabling `compat/poll/`, `compat/regex/`,
`compat/win32mmap.c` & friends depending on the current platform... To
put it mildly: it was a major mess.
Instead, this patch makes minimal changes to the top-level `Makefile` so
that the bits in `contrib/scalar/` can be compiled and linked, and
adds a `contrib/scalar/Makefile` that uses the top-level `Makefile` in a
most minimal way to do the actual compiling.
Note: With this commit, we only establish the infrastructure, no
Scalar functionality is implemented yet; We will do that incrementally
over the next few commits.
Signed-off-by: Johannes Schindelin <redacted>
---
Makefile | 8 ++++++++
contrib/scalar/.gitignore | 2 ++
contrib/scalar/Makefile | 34 ++++++++++++++++++++++++++++++++++
contrib/scalar/scalar.c | 36 ++++++++++++++++++++++++++++++++++++
4 files changed, 80 insertions(+)
create mode 100644 contrib/scalar/.gitignore
create mode 100644 contrib/scalar/Makefile
create mode 100644 contrib/scalar/scalar.c
@@ -0,0 +1,34 @@+QUIET_SUBDIR0=+$(MAKE)-C# space to separate -C and subdir+QUIET_SUBDIR1=++ifneq ($(findstring s,$(MAKEFLAGS)),s)+ifndef V+QUIET_SUBDIR0=+@subdir=+QUIET_SUBDIR1=;$(NO_SUBDIR)echo' 'SUBDIR$$subdir;\+$(MAKE)$(PRINT_DIR)-C$$subdir+else+exportV+endif+endif++all:++include ../../config.mak.uname+-include ../../config.mak.autogen+-include ../../config.mak++TARGETS=scalar$(X)scalar.o+GITLIBS=../../common-main.o../../libgit.a../../xdiff/lib.a++all:scalar$X++$(GITLIBS):+$(QUIET_SUBDIR0)../..$(QUIET_SUBDIR1)$(subst../../,,$@)++$(TARGETS):$(GITLIBS)scalar.c+$(QUIET_SUBDIR0)../..$(QUIET_SUBDIR1)$(patsubst%,contrib/scalar/%,$@)++clean:+$(RM)$(TARGETS)++.PHONY:allcleanFORCE
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-09-03 17:54:57
From: Johannes Schindelin <redacted>
To test the Scalar command, create a test script in contrib/scalar/t
that is executed as `make -C contrib/scalar test`. Since Scalar has no
meaningful capabilities yet, the only test is rather simple. We will add
more tests in subsequent commits that introduce corresponding, new
functionality.
Note: this test script is intended to test `scalar` only lightly, even
after all of the functionality is implemented.
A more comprehensive functional (or: integration) test suite can be
found at https://github.com/microsoft/scalar; It is used in the workflow
https://github.com/microsoft/git/blob/HEAD/.github/workflows/scalar-functional-tests.yml
in Microsoft's Git fork. This test suite performs end-to-end tests with
a real remote repository, and is run as part of the regular CI builds.
Since those tests require some functionality supported only by
Microsoft's Git fork ("GVFS protocol"), there is no intention to port
that fuller test suite to `contrib/scalar/`.
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/Makefile | 17 +++++--
contrib/scalar/t/Makefile | 78 ++++++++++++++++++++++++++++++++
contrib/scalar/t/t9099-scalar.sh | 17 +++++++
3 files changed, 109 insertions(+), 3 deletions(-)
create mode 100644 contrib/scalar/t/Makefile
create mode 100755 contrib/scalar/t/t9099-scalar.sh
@@ -21,7 +22,7 @@ include ../../config.mak.unameTARGETS=scalar$(X)scalar.oGITLIBS=../../common-main.o../../libgit.a../../xdiff/lib.a-all:scalar$X+all:scalar$X ../../bin-wrappers/scalar$(GITLIBS):$(QUIET_SUBDIR0)../..$(QUIET_SUBDIR1)$(subst../../,,$@)
@@ -30,9 +31,19 @@ $(TARGETS): $(GITLIBS) scalar.c$(QUIET_SUBDIR0)../..$(QUIET_SUBDIR1)$(patsubst%,contrib/scalar/%,$@)clean:-$(RM)$(TARGETS)+$(RM)$(TARGETS)../../bin-wrappers/scalar$(RM)scalar.1scalar.htmlscalar.xml+../../bin-wrappers/scalar:../../wrap-for-bin.shMakefile+@mkdir-p../../bin-wrappers+$(QUIET_GEN)sed-e'1s|#!.*/sh|#!$(SHELL_PATH_SQ)|'\+-e's|@@BUILD_DIR@@|$(shell cd ../.. && pwd)|'\+-e's|@@PROG@@|contrib/scalar/scalar$(X)|'<$<>$@&&\+chmod+x$@++test:all+$(MAKE)-Ct+docs:scalar.htmlscalar.1scalar.html:|scalar.1 # prevent them from trying to build `doc.dep` in parallel
@@ -0,0 +1,17 @@+#!/bin/sh++test_description='test the `scalar` command'++TEST_DIRECTORY=$PWD/../../../t+exportTEST_DIRECTORY++# Make it work with --no-bin-wrappers+PATH=$PWD/..:$PATH++.../../../t/test-lib.sh++test_expect_success'scalar shows a usage''+test_expect_code129scalar-h+'++test_done
@@ -30,5 +31,16 @@ $(TARGETS): $(GITLIBS) scalar.cclean:$(RM)$(TARGETS)+$(RM)scalar.1scalar.htmlscalar.xml-.PHONY:allcleanFORCE+docs:scalar.htmlscalar.1++scalar.html:|scalar.1 # prevent them from trying to build `doc.dep` in parallel++scalar.html scalar.1:scalar.txt+$(QUIET_SUBDIR0)../../Documentation$(QUIET_SUBDIR1)\+MAN_TXT=../contrib/scalar/scalar.txt\+../contrib/scalar/$@+$(QUIET)testscalar.1!="$@"||mv../../Documentation/$@.++.PHONY:allcleandocsFORCE
@@ -0,0 +1,38 @@+scalar(1)+=========++NAME+----+scalar - an opinionated repository management tool++SYNOPSIS+--------+[verse]+scalar <command> [<options>]++DESCRIPTION+-----------++Scalar is an opinionated repository management tool. By creating new+repositories or registering existing repositories with Scalar, your Git+experience will speed up. Scalar sets advanced Git config settings,+maintains your repositories in the background, and helps reduce data sent+across the network.++An important Scalar concept is the enlistment: this is the top-level directory+of the project. It usually contains the subdirectory `src/` which is a Git+worktree. This encourages the separation between tracked files (inside `src/`)+and untracked files, such as build artifacts (outside `src/`). When registering+an existing Git worktree with Scalar whose name is not `src`, the enlistment+will be identical to the worktree.++The `scalar` command implements various subcommands, and different options+depending on the subcommand.++SEE ALSO+--------+linkgit:git-maintenance[1].++Scalar+---+Associated with the linkgit:git[1] suite
From: Derrick Stolee via GitGitGadget <hidden> Date: 2021-09-03 17:54:59
From: Derrick Stolee <redacted>
Let's start implementing the `register` command. With this commit,
recommended settings are configured upon `scalar register`, and Git's
background maintenance is started.
The recommended config settings may very well change in the future. For
example, once the built-in FSMonitor is available, we will want to
enable it upon `scalar register`. For that reason, we explicitly support
running `scalar register` in an already-registered enlistment.
Co-authored-by: Victoria Dye [off-list ref]
Signed-off-by: Derrick Stolee <redacted>
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 255 ++++++++++++++++++++++++++++++++++++++
contrib/scalar/scalar.txt | 18 ++-
2 files changed, 272 insertions(+), 1 deletion(-)
@@ -5,11 +5,266 @@#include"cache.h"#include"gettext.h"#include"parse-options.h"+#include"config.h"+#include"run-command.h"++/*+*Removethedeepestsubdirectoryintheprovidedpathstring.Pathmustnot+*includeatrailingpathseparator.Returns1ifparentdirectoryfound,+*otherwise0.+*/+staticintstrbuf_parent_directory(structstrbuf*buf)+{+size_tlen=buf->len;+size_toffset=offset_1st_component(buf->buf);+char*path_sep=find_last_dir_sep(buf->buf+offset);+strbuf_setlen(buf,path_sep?path_sep-buf->buf:offset);++returnbuf->len<len;+}++staticvoidsetup_enlistment_directory(intargc,constchar**argv,+constchar*const*usagestr,+conststructoption*options,+structstrbuf*enlistment_root)+{+structstrbufpath=STRBUF_INIT;+char*root;+intenlistment_found=0;++if(startup_info->have_repository)+BUG("gitdir already set up?!?");++if(argc>1)+usage_with_options(usagestr,options);++/* find the worktree, determine its corresponding root */+if(argc==1)+strbuf_add_absolute_path(&path,argv[0]);+elseif(strbuf_getcwd(&path)<0)+die(_("need a working directory"));++strbuf_trim_trailing_dir_sep(&path);+do{+constsize_tlen=path.len;++/* check if currently in enlistment root with src/ workdir */+strbuf_addstr(&path,"/src/.git");+if(is_git_directory(path.buf)){+strbuf_strip_suffix(&path,"/.git");++if(enlistment_root)+strbuf_add(enlistment_root,path.buf,len);++enlistment_found=1;+break;+}++/* reset to original path */+strbuf_setlen(&path,len);++/* check if currently in workdir */+strbuf_addstr(&path,"/.git");+if(is_git_directory(path.buf)){+strbuf_setlen(&path,len);++if(enlistment_root){+/*+*Iftheworktree'sdirectory'snameis`src`,theenlistmentisthe+*parentdirectory,otherwiseitisidenticaltotheworktree.+*/+root=strip_path_suffix(path.buf,"src");+strbuf_addstr(enlistment_root,root?root:path.buf);+free(root);+}++enlistment_found=1;+break;+}++strbuf_setlen(&path,len);+}while(strbuf_parent_directory(&path));++if(!enlistment_found)+die(_("could not find enlistment root"));++if(chdir(path.buf)<0)+die_errno(_("could not switch to '%s'"),path.buf);++strbuf_release(&path);+setup_git_directory();+}++staticintrun_git(constchar*arg,...)+{+structstrvecargv=STRVEC_INIT;+va_listargs;+constchar*p;+intres;++va_start(args,arg);+strvec_push(&argv,arg);+while((p=va_arg(args,constchar*)))+strvec_push(&argv,p);+va_end(args);++res=run_command_v_opt(argv.v,RUN_GIT_CMD);++strvec_clear(&argv);+returnres;+}++staticintset_recommended_config(void)+{+struct{+constchar*key;+constchar*value;+}config[]={+{"am.keepCR","true"},+{"core.FSCache","true"},+{"core.multiPackIndex","true"},+{"core.preloadIndex","true"},+#ifndef WIN32+{"core.untrackedCache","true"},+#else+/*+*Unfortunately,Scalar'sFunctionalTestsdemonstrated+*thattheuntrackedcachefeatureisunreliableonWindows+*(whichisabummerbecausethatplatformwouldbenefitthe+*mostfromit).Forsomereason,freshlycreatedfilesseem+*nottoupdatethedirectory's`lastModified`time+*immediately,buttheuntrackedcachewouldneedtorelyon+*that.+*+*Therefore,withasadheart,wedisablethisveryuseful+*featureonWindows.+*/+{"core.untrackedCache","false"},+#endif+{"core.logAllRefUpdates","true"},+{"credential.https://dev.azure.com.useHttpPath","true"},+{"credential.validate","false"},/* GCM4W-only */+{"gc.auto","0"},+{"gui.GCWarning","false"},+{"index.threads","true"},+{"index.version","4"},+{"merge.stat","false"},+{"merge.renames","false"},+{"pack.useBitmaps","false"},+{"pack.useSparse","true"},+{"receive.autoGC","false"},+{"reset.quiet","true"},+{"feature.manyFiles","false"},+{"feature.experimental","false"},+{"fetch.unpackLimit","1"},+{"fetch.writeCommitGraph","false"},+#ifdef WIN32+{"http.sslBackend","schannel"},+#endif+{"status.aheadBehind","false"},+{"commitGraph.generationVersion","1"},+{"core.autoCRLF","false"},+{"core.safeCRLF","false"},+{NULL,NULL},+};+inti;+char*value;++for(i=0;config[i].key;i++){+if(git_config_get_string(config[i].key,&value)){+trace2_data_string("scalar",the_repository,config[i].key,"created");+if(git_config_set_gently(config[i].key,+config[i].value)<0)+returnerror(_("could not configure %s=%s"),+config[i].key,config[i].value);+}else{+trace2_data_string("scalar",the_repository,config[i].key,"exists");+free(value);+}+}++/*+*The`log.excludeDecoration`settingisspecialbecauseitallows+*formultiplevalues.+*/+if(git_config_get_string("log.excludeDecoration",&value)){+trace2_data_string("scalar",the_repository,+"log.excludeDecoration","created");+if(git_config_set_multivar_gently("log.excludeDecoration",+"refs/prefetch/*",+CONFIG_REGEX_NONE,0))+returnerror(_("could not configure "+"log.excludeDecoration"));+}else{+trace2_data_string("scalar",the_repository,+"log.excludeDecoration","exists");+free(value);+}++return0;+}++staticintstart_maintenance(void)+{+returnrun_git("maintenance","start",NULL);+}++staticintadd_enlistment(void)+{+intres;++if(!the_repository->worktree)+die(_("Scalar enlistments require a worktree"));++res=run_git("config","--global","--get","--fixed-value",+"scalar.repo",the_repository->worktree,NULL);++/*+*Ifthesettingisalreadythere,thendonothing.+*/+if(!res)+return0;++returnrun_git("config","--global","--add",+"scalar.repo",the_repository->worktree,NULL);+}++staticintregister_dir(void)+{+intres=add_enlistment();++if(!res)+res=set_recommended_config();++if(!res)+res=start_maintenance();++returnres;+}++staticintcmd_register(intargc,constchar**argv)+{+structoptionoptions[]={+OPT_END(),+};+constchar*constusage[]={+N_("scalar register [<enlistment>]"),+NULL+};++argc=parse_options(argc,argv,NULL,options,+usage,0);++setup_enlistment_directory(argc,argv,usage,options,NULL);++returnregister_dir();+}staticstruct{constchar*name;int(*fn)(int,constchar**);}builtins[]={+{"register",cmd_register},{NULL,NULL},};
@@ -29,6 +29,22 @@ will be identical to the worktree. The `scalar` command implements various subcommands, and different options depending on the subcommand.+COMMANDS+--------++Register+~~~~~~~~++register [<enlistment>]::+ Adds the enlistment's repository to the list of registered repositories+ and starts background maintenance. If `<enlistment>` is not provided,+ then the enlistment associated with the current working directory is+ registered.+++Note: when this subcommand is called in a worktree that is called `src/`, its+parent directory is considered to be the Scalar enlistment. If the worktree is+_not_ called `src/`, it itself will be considered to be the Scalar enlistment.+ SEE ALSO -------- linkgit:git-maintenance[1].
From: Derrick Stolee via GitGitGadget <hidden> Date: 2021-09-03 17:55:01
From: Derrick Stolee <redacted>
Just like `scalar register` starts the scheduled background maintenance,
`scalar unregister` stops it. Note that we use `git maintenance start`
in `scalar register`, but we do not use `git maintenance stop` in
`scalar unregister`: this would stop maintenance for _all_ repositories,
not just for the one we want to unregister.
The `unregister` command also removes the corresponding entry from the
`[scalar]` section in the global Git config.
Co-authored-by: Victoria Dye [off-list ref]
Signed-off-by: Derrick Stolee <redacted>
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 50 ++++++++++++++++++++++++++++++++-------
contrib/scalar/scalar.txt | 8 +++++++
2 files changed, 50 insertions(+), 8 deletions(-)
@@ -45,6 +46,13 @@ Note: when this subcommand is called in a worktree that is called `src/`, its parent directory is considered to be the Scalar enlistment. If the worktree is _not_ called `src/`, it itself will be considered to be the Scalar enlistment.+Unregister+~~~~~~~~~~++unregister [<enlistment>]::+ Remove the specified repository from the list of repositories+ registered with Scalar and stop the scheduled background maintenance.+ SEE ALSO -------- linkgit:git-maintenance[1].
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-09-03 17:55:02
From: Johannes Schindelin <redacted>
When a user deleted an enlistment manually, let's be generous and
_still_ unregister it.
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 46 ++++++++++++++++++++++++++++++++
contrib/scalar/t/t9099-scalar.sh | 15 +++++++++++
2 files changed, 61 insertions(+)
From: Derrick Stolee via GitGitGadget <hidden> Date: 2021-09-03 17:55:07
From: Derrick Stolee <redacted>
The produced list simply consists of those repositories registered under
the multi-valued `scalar.repo` config setting in the user's Git config.
Signed-off-by: Derrick Stolee <redacted>
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 11 +++++++++++
contrib/scalar/scalar.txt | 12 +++++++++++-
2 files changed, 22 insertions(+), 1 deletion(-)
@@ -257,6 +257,16 @@ static int unregister_dir(void)returnres;}+staticintcmd_list(intargc,constchar**argv)+{+if(argc!=1)+die(_("`scalar list` does not take arguments"));++if(run_git("config","--global","--get-all","scalar.repo",NULL)<0)+return-1;+return0;+}+staticintcmd_register(intargc,constchar**argv){structoptionoptions[]={
@@ -28,11 +29,20 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment will be identical to the worktree. The `scalar` command implements various subcommands, and different options-depending on the subcommand.+depending on the subcommand. With the exception of `list`, all subcommands+expect to be run in an enlistment. COMMANDS --------+List+~~~~++list::+ To see which repositories are currently registered by the service, run+ `scalar list`. This subcommand does not need to be run inside a Scalar+ enlistment.+ Register ~~~~~~~~
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-09-03 17:55:08
From: Johannes Schindelin <redacted>
This implements Scalar's opinionated `clone` command: it tries to use a
partial clone and sets up a sparse checkout by default. In contrast to
`git clone`, `scalar clone` sets up the worktree in the `src/`
subdirectory, to encourage a separation between the source files and the
build output (which helps Git tremendously because it avoids untracked
files that have to be specifically ignored when refreshing the index).
Also, it registers the repository for regular, scheduled maintenance,
and configures a flurry of configuration settings based on the
experience and experiments of the Microsoft Windows and the Microsoft
Office development teams.
Note: since the `scalar clone` command is by far the most commonly
called `scalar` subcommand, we document it at the top of the manual
page.
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 201 +++++++++++++++++++++++++++++++
contrib/scalar/scalar.txt | 35 +++++-
contrib/scalar/t/t9099-scalar.sh | 32 +++++
3 files changed, 263 insertions(+), 5 deletions(-)
@@ -257,6 +258,205 @@ static int unregister_dir(void)returnres;}+/* printf-style interface, expects `<key>=<value>` argument */+staticintset_config(constchar*fmt,...)+{+structstrbufbuf=STRBUF_INIT;+char*value;+intres;+va_listargs;++va_start(args,fmt);+strbuf_vaddf(&buf,fmt,args);+va_end(args);++value=strchr(buf.buf,'=');+if(value)+*(value++)='\0';+res=git_config_set_gently(buf.buf,value);+strbuf_release(&buf);++returnres;+}++staticchar*remote_default_branch(constchar*url)+{+structchild_processcp=CHILD_PROCESS_INIT;+structstrbufout=STRBUF_INIT;++cp.git_cmd=1;+strvec_pushl(&cp.args,"ls-remote","--symref",url,"HEAD",NULL);+if(!pipe_command(&cp,NULL,0,&out,0,NULL,0)){+constchar*line=out.buf;++while(*line){+constchar*eol=strchrnul(line,'\n'),*p;+size_tlen=eol-line;+char*branch;++if(!skip_prefix(line,"ref: ",&p)||+!strip_suffix_mem(line,&len,"\tHEAD")){+line=eol+(*eol=='\n');+continue;+}++eol=line+len;+if(skip_prefix(p,"refs/heads/",&p)){+branch=xstrndup(p,eol-p);+strbuf_release(&out);+returnbranch;+}++error(_("remote HEAD is not a branch: '%.*s'"),+(int)(eol-p),p);+strbuf_release(&out);+returnNULL;+}+}+warning(_("failed to get default branch name from remote; "+"using local default"));+strbuf_reset(&out);++child_process_init(&cp);+cp.git_cmd=1;+strvec_pushl(&cp.args,"symbolic-ref","--short","HEAD",NULL);+if(!pipe_command(&cp,NULL,0,&out,0,NULL,0)){+strbuf_trim(&out);+returnstrbuf_detach(&out,NULL);+}++strbuf_release(&out);+error(_("failed to get default branch name"));+returnNULL;+}++staticintcmd_clone(intargc,constchar**argv)+{+constchar*branch=NULL;+intfull_clone=0;+structoptionclone_options[]={+OPT_STRING('b',"branch",&branch,N_("<branch>"),+N_("branch to checkout after clone")),+OPT_BOOL(0,"full-clone",&full_clone,+N_("when cloning, create full working directory")),+OPT_END(),+};+constchar*constclone_usage[]={+N_("scalar clone [<options>] [--] <repo> [<dir>]"),+NULL+};+constchar*url;+char*enlistment=NULL,*dir=NULL;+structstrbufbuf=STRBUF_INIT;+intres;++argc=parse_options(argc,argv,NULL,clone_options,clone_usage,0);++if(argc==2){+url=argv[0];+enlistment=xstrdup(argv[1]);+}elseif(argc==1){+url=argv[0];++strbuf_addstr(&buf,url);+/* Strip trailing slashes, if any */+while(buf.len>0&&is_dir_sep(buf.buf[buf.len-1]))+strbuf_setlen(&buf,buf.len-1);+/* Strip suffix `.git`, if any */+strbuf_strip_suffix(&buf,".git");++enlistment=find_last_dir_sep(buf.buf);+if(!enlistment){+die(_("cannot deduce worktree name from '%s'"),url);+}+enlistment=xstrdup(enlistment+1);+}else{+usage_msg_opt(_("You must specify a repository to clone."),+clone_usage,clone_options);+}++if(is_directory(enlistment))+die(_("directory '%s' exists already"),enlistment);++dir=xstrfmt("%s/src",enlistment);++strbuf_reset(&buf);+if(branch)+strbuf_addf(&buf,"init.defaultBranch=%s",branch);+else{+char*b=repo_default_branch_name(the_repository,1);+strbuf_addf(&buf,"init.defaultBranch=%s",b);+free(b);+}++if((res=run_git("-c",buf.buf,"init","--",dir,NULL)))+gotocleanup;++if(chdir(dir)<0){+res=error_errno(_("could not switch to '%s'"),dir);+gotocleanup;+}++setup_git_directory();++/* common-main already logs `argv` */+trace2_def_repo(the_repository);++if(!branch&&!(branch=remote_default_branch(url))){+res=error(_("failed to get default branch for '%s'"),url);+gotocleanup;+}++if(set_config("remote.origin.url=%s",url)||+set_config("remote.origin.fetch="+"+refs/heads/*:refs/remotes/origin/*")||+set_config("remote.origin.promisor=true")||+set_config("remote.origin.partialCloneFilter=blob:none")){+res=error(_("could not configure remote in '%s'"),dir);+gotocleanup;+}++if(!full_clone&&+(res=run_git("sparse-checkout","init","--cone",NULL)))+gotocleanup;++if(set_recommended_config())+returnerror(_("could not configure '%s'"),dir);++if((res=run_git("fetch","--quiet","origin",NULL))){+warning(_("partial clone failed; attempting full clone"));++if(set_config("remote.origin.promisor")||+set_config("remote.origin.partialCloneFilter")){+res=error(_("could not configure for full clone"));+gotocleanup;+}++if((res=run_git("fetch","--quiet","origin",NULL)))+gotocleanup;+}++if((res=set_config("branch.%s.remote=origin",branch)))+gotocleanup;+if((res=set_config("branch.%s.merge=refs/heads/%s",+branch,branch)))+gotocleanup;++strbuf_reset(&buf);+strbuf_addf(&buf,"origin/%s",branch);+res=run_git("checkout","-f","-t",buf.buf,NULL);+if(res)+gotocleanup;++res=register_dir();++cleanup:+free(enlistment);+free(dir);+strbuf_release(&buf);+returnres;+}+staticintcmd_list(intargc,constchar**argv){if(argc!=1)
@@ -29,19 +30,43 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment will be identical to the worktree. The `scalar` command implements various subcommands, and different options-depending on the subcommand. With the exception of `list`, all subcommands-expect to be run in an enlistment.+depending on the subcommand. With the exception of `clone` and `list`, all+subcommands expect to be run in an enlistment. COMMANDS --------+Clone+~~~~~++clone [<options>] <url> [<enlistment>]::+ Clones the specified repository, similar to linkgit:git-clone[1]. By+ default, only commit and tree objects are cloned. Once finished, the+ worktree is located at `<enlistment>/src`.+++The sparse-checkout feature is enabled (except when run with `--full-clone`)+and the only files present are those in the top-level directory. Use+`git sparse-checkout set` to expand the set of directories you want to see,+or `git sparse-checkout disable` to expand to all files (see+linkgit:git-sparse-checkout[1] for more details). You can explore the+subdirectories outside your sparse-checkout by using `git ls-tree HEAD`.++-b <name>::+--branch <name>::+ Instead of checking out the branch pointed to by the cloned+ repository's HEAD, check out the `<name>` branch instead.++--[no-]full-clone::+ A sparse-checkout is initialized by default. This behavior can be+ turned off via `--full-clone`.+ List ~~~~ list:: To see which repositories are currently registered by the service, run- `scalar list`. This subcommand does not need to be run inside a Scalar- enlistment.+ `scalar list`. This subcommand, like `clone`, does not need to be run+ inside a Scalar enlistment. Register ~~~~~~~~
@@ -65,7 +90,7 @@ unregister [<enlistment>]:: SEE ALSO ---------linkgit:git-maintenance[1].+linkgit:git-clone[1], linkgit:git-maintenance[1]. Scalar ---
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-09-03 17:55:09
From: Johannes Schindelin <redacted>
Just like `git clone`, the `scalar clone` command now also offers to
restrict the clone to a single branch.
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 9 +++++++--
contrib/scalar/scalar.txt | 12 +++++++++++-
contrib/scalar/t/t9099-scalar.sh | 6 +++++-
3 files changed, 23 insertions(+), 4 deletions(-)
@@ -333,12 +333,15 @@ static char *remote_default_branch(const char *url)staticintcmd_clone(intargc,constchar**argv){constchar*branch=NULL;-intfull_clone=0;+intfull_clone=0,single_branch=0;structoptionclone_options[]={OPT_STRING('b',"branch",&branch,N_("<branch>"),N_("branch to checkout after clone")),OPT_BOOL(0,"full-clone",&full_clone,N_("when cloning, create full working directory")),+OPT_BOOL(0,"single-branch",&single_branch,+N_("only download metadata for the branch that will "+"be checked out")),OPT_END(),};constchar*constclone_usage[]={
@@ -409,7 +412,9 @@ static int cmd_clone(int argc, const char **argv)if(set_config("remote.origin.url=%s",url)||set_config("remote.origin.fetch="-"+refs/heads/*:refs/remotes/origin/*")||+"+refs/heads/%s:refs/remotes/origin/%s",+single_branch?branch:"*",+single_branch?branch:"*")||set_config("remote.origin.promisor=true")||set_config("remote.origin.partialCloneFilter=blob:none")){res=error(_("could not configure remote in '%s'"),dir);
@@ -56,6 +56,16 @@ subdirectories outside your sparse-checkout by using `git ls-tree HEAD`. Instead of checking out the branch pointed to by the cloned repository's HEAD, check out the `<name>` branch instead.+--[no-]single-branch::+ Clone only the history leading to the tip of a single branch, either+ specified by the `--branch` option or the primary branch remote's+ `HEAD` points at.+++Further fetches into the resulting repository will only update the+remote-tracking branch for the branch this option was used for the initial+cloning. If the HEAD at the remote did not point at any branch when+`--single-branch` clone was made, no remote-tracking branch is created.+ --[no-]full-clone:: A sparse-checkout is initialized by default. This behavior can be turned off via `--full-clone`.
From: Derrick Stolee via GitGitGadget <hidden> Date: 2021-09-03 17:55:11
From: Derrick Stolee <redacted>
Note: this subcommand is provided primarily for backwards-compatibility,
for existing Scalar uses. It is mostly just a shim for `git
maintenance`, mapping task names from the way Scalar called them to the
way Git calls them.
The reason why those names differ? The background maintenance was first
implemented in Scalar, and when it was contributed as a patch series
implementing the `git maintenance` command, reviewers suggested better
names, those suggestions were accepted before the patches were
integrated into core Git.
Signed-off-by: Derrick Stolee <redacted>
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 64 +++++++++++++++++++++++++++++++++++++++
contrib/scalar/scalar.txt | 19 ++++++++++++
2 files changed, 83 insertions(+)
@@ -490,6 +490,69 @@ static int cmd_register(int argc, const char **argv)returnregister_dir();}+staticintcmd_run(intargc,constchar**argv)+{+structoptionoptions[]={+OPT_END(),+};+struct{+constchar*arg,*task;+}tasks[]={+{"config",NULL},+{"commit-graph","commit-graph"},+{"fetch","prefetch"},+{"loose-objects","loose-objects"},+{"pack-files","incremental-repack"},+{NULL,NULL}+};+structstrbufbuf=STRBUF_INIT;+constchar*usagestr[]={NULL,NULL};+inti;++strbuf_addstr(&buf,N_("scalar run <task> [<enlistment>]\nTasks:\n"));+for(i=0;tasks[i].arg;i++)+strbuf_addf(&buf,"\t%s\n",tasks[i].arg);+usagestr[0]=buf.buf;++argc=parse_options(argc,argv,NULL,options,+usagestr,0);++if(argc==0)+usage_with_options(usagestr,options);++if(!strcmp("all",argv[0]))+i=-1;+else{+for(i=0;tasks[i].arg&&strcmp(tasks[i].arg,argv[0]);i++)+;/* keep looking for the task */++if(i>0&&!tasks[i].arg){+error(_("no such task: '%s'"),argv[0]);+usage_with_options(usagestr,options);+}+}++argc--;+argv++;+setup_enlistment_directory(argc,argv,usagestr,options,NULL);+strbuf_release(&buf);++if(i==0)+returnregister_dir();++if(i>0)+returnrun_git("maintenance","run",+"--task",tasks[i].task,NULL);++if(register_dir())+return-1;+for(i=1;tasks[i].arg;i++)+if(run_git("maintenance","run",+"--task",tasks[i].task,NULL))+return-1;+return0;+}+staticintremove_deleted_enlistment(structstrbuf*path){intres=0;
@@ -98,6 +99,24 @@ unregister [<enlistment>]:: Remove the specified repository from the list of repositories registered with Scalar and stop the scheduled background maintenance.+Run+~~~++scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]::+ Run the given maintenance task (or all tasks, if `all` was specified).+ Except for `all` and `config`, this subcommand simply hands off to+ linkgit:git-maintenance[1] (mapping `fetch` to `prefetch` and+ `pack-files` to `incremental-repack`).+++These tasks are run automatically as part of the scheduled maintenance,+as soon as the repository is registered with Scalar. It should therefore+not be necessary to run this subcommand manually.+++The `config` task is specific to Scalar and configures all those+opinionated default settings that make Git work more efficiently with+large repositories. As this task is run as part of `scalar clone`+automatically, explicit invocations of this task are rarely needed.+ SEE ALSO -------- linkgit:git-clone[1], linkgit:git-maintenance[1].
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-09-03 17:55:11
From: Johannes Schindelin <redacted>
After a Scalar upgrade, it can come in really handy if there is an easy
way to reconfigure all Scalar enlistments. This new option offers this
functionality.
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 61 ++++++++++++++++++++++++++++++--
contrib/scalar/scalar.txt | 10 ++++--
contrib/scalar/t/t9099-scalar.sh | 3 ++
3 files changed, 68 insertions(+), 6 deletions(-)
@@ -494,22 +494,77 @@ static int cmd_register(int argc, const char **argv)returnregister_dir();}+staticintget_scalar_repos(constchar*key,constchar*value,void*data)+{+structstring_list*list=data;++if(!strcmp(key,"scalar.repo"))+string_list_append(list,value);++return0;+}+staticintcmd_reconfigure(intargc,constchar**argv){+intall=0;structoptionoptions[]={+OPT_BOOL('a',"all",&all,+N_("reconfigure all registered enlistments")),OPT_END(),};constchar*constusage[]={-N_("scalar reconfigure [<enlistment>]"),+N_("scalar reconfigure [--all | <enlistment>]"),NULL};+structstring_listscalar_repos=STRING_LIST_INIT_DUP;+inti,res=0;+structrepositoryr={NULL};+structstrbufcommondir=STRBUF_INIT,gitdir=STRBUF_INIT;argc=parse_options(argc,argv,NULL,options,usage,0);-setup_enlistment_directory(argc,argv,usage,options,NULL);+if(!all){+setup_enlistment_directory(argc,argv,usage,options,NULL);++returnset_recommended_config(1);+}++if(argc>0)+usage_msg_opt(_("--all or <enlistment>, but not both"),+usage,options);++git_config(get_scalar_repos,&scalar_repos);-returnset_recommended_config(1);+for(i=0;i<scalar_repos.nr;i++){+constchar*dir=scalar_repos.items[i].string;++strbuf_reset(&commondir);+strbuf_reset(&gitdir);++if(chdir(dir)<0){+warning_errno(_("could not switch to '%s'"),dir);+res=-1;+}elseif(discover_git_directory(&commondir,&gitdir)<0){+warning_errno(_("git repository gone in '%s'"),dir);+res=-1;+}else{+git_config_clear();++the_repository=&r;+r.commondir=commondir.buf;+r.gitdir=gitdir.buf;++if(set_recommended_config(1)<0)+res=-1;+}+}++string_list_clear(&scalar_repos,1);+strbuf_release(&commondir);+strbuf_release(&gitdir);++returnres;}staticintcmd_run(intargc,constchar**argv)
@@ -32,8 +32,8 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment will be identical to the worktree. The `scalar` command implements various subcommands, and different options-depending on the subcommand. With the exception of `clone` and `list`, all-subcommands expect to be run in an enlistment.+depending on the subcommand. With the exception of `clone`, `list` and+`reconfigure --all`, all subcommands expect to be run in an enlistment. COMMANDS --------
@@ -125,6 +125,10 @@ After a Scalar upgrade, or when the configuration of a Scalar enlistment was somehow corrupted or changed by mistake, this subcommand allows to reconfigure the enlistment.+With the `--all` option, all enlistments currently registered with Scalar+will be reconfigured. This option is meant to to be run every time after+Scalar is upgraded.+ SEE ALSO -------- linkgit:git-clone[1], linkgit:git-maintenance[1].
From: Matthew John Cheetham via GitGitGadget <hidden> Date: 2021-09-03 17:55:13
From: Matthew John Cheetham <redacted>
Delete an enlistment by first unregistering the repository and then
deleting the enlistment directory (usually the directory containing the
worktree `src/` directory).
On Windows, if the current directory is inside the enlistment's
directory, change to the parent of the enlistment directory, to allow us
to delete the enlistment (directories used by processes e.g. as current
working directories cannot be deleted on Windows).
Co-authored-by: Victoria Dye [off-list ref]
Signed-off-by: Matthew John Cheetham <redacted>
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 55 ++++++++++++++++++++++++++++++++
contrib/scalar/scalar.txt | 8 +++++
contrib/scalar/t/t9099-scalar.sh | 9 ++++++
3 files changed, 72 insertions(+)
@@ -129,6 +130,13 @@ With the `--all` option, all enlistments currently registered with Scalar will be reconfigured. This option is meant to to be run every time after Scalar is upgraded.+Delete+~~~~~~++delete <enlistment>::+ This subcommand lets you delete an existing Scalar enlistment from your+ local file system, unregistering the repository.+ SEE ALSO -------- linkgit:git-clone[1], linkgit:git-maintenance[1].
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-09-03 17:55:14
From: Johannes Schindelin <redacted>
This comes in handy during Scalar upgrades, or when config settings were
messed up by mistake.
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 79 +++++++++++++++++++++-----------
contrib/scalar/scalar.txt | 8 ++++
contrib/scalar/t/t9099-scalar.sh | 8 ++++
3 files changed, 67 insertions(+), 28 deletions(-)
@@ -117,6 +118,13 @@ opinionated default settings that make Git work more efficiently with large repositories. As this task is run as part of `scalar clone` automatically, explicit invocations of this task are rarely needed.+Reconfigure+~~~~~~~~~~~++After a Scalar upgrade, or when the configuration of a Scalar enlistment+was somehow corrupted or changed by mistake, this subcommand allows to+reconfigure the enlistment.+ SEE ALSO -------- linkgit:git-clone[1], linkgit:git-maintenance[1].
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-09-03 17:55:17
From: Johannes Schindelin <redacted>
The .NET version of Scalar has a `version` command. This was necessary
because it was versioned independently of Git.
Since Scalar is now tightly coupled with Git, it does not make sense for
them to show different versions. Therefore, it shows the same output as
`git version`. For backwards-compatibility with the .NET version,
`scalar version` prints to `stderr`, though (`git version` prints to
`stdout` instead).
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 39 +++++++++++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-09-03 17:55:17
From: Johannes Schindelin <redacted>
The `git` executable has these two very useful options:
-C <directory>:
switch to the specified directory before performing any actions
-c <key>=<value>:
temporarily configure this setting for the duration of the
specified scalar subcommand
With this commit, we teach the `scalar` executable the same trick.
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 22 +++++++++++++++++++++-
contrib/scalar/scalar.txt | 10 ++++++++++
2 files changed, 31 insertions(+), 1 deletion(-)
@@ -806,6 +806,25 @@ int cmd_main(int argc, const char **argv)structstrbufscalar_usage=STRBUF_INIT;inti;+while(argc>1&&*argv[1]=='-'){+if(!strcmp(argv[1],"-C")){+if(argc<3)+die(_("-C requires a <directory>"));+if(chdir(argv[2])<0)+die_errno(_("could not change to '%s'"),+argv[2]);+argc-=2;+argv+=2;+}elseif(!strcmp(argv[1],"-c")){+if(argc<3)+die(_("-c requires a <key>=<value> argument"));+git_config_push_parameter(argv[2]);+argc-=2;+argv+=2;+}else+break;+}+if(argc>1){argv++;argc--;
@@ -36,6 +36,16 @@ The `scalar` command implements various subcommands, and different options depending on the subcommand. With the exception of `clone`, `list` and `reconfigure --all`, all subcommands expect to be run in an enlistment.+The following options can be specified _before_ the subcommand:++-C <directory>::+ Before running the subcommand, change the working directory. This+ option imitates the same option of linkgit:git[1].++-c <key>=<value>::+ For the duration of running the specified subcommand, configure this+ setting. This option imitates the same option of linkgit:git[1].+ COMMANDS --------
On 04/09/21 00.54, Derrick Stolee via GitGitGadget wrote:
+List
+~~~~
+
+list::
+ To see which repositories are currently registered by the service, run
+ `scalar list`. This subcommand does not need to be run inside a Scalar
+ enlistment.
+
I think the man-page-style wording should be:
list::
List enlistments that are currently registered by Scalar. This
subcommand does not need to be run inside an enlistment.
--
An old man doll... just what I always wanted! - Clara
Hi Ævar,
On Tue, 31 Aug 2021, Ævar Arnfjörð Bjarmason wrote:
quoted
On Mon, Aug 30 2021, Johannes Schindelin via GitGitGadget wrote:
quoted
This comes in handy during Scalar upgrades, or when config settings were
messed up by mistake.
quoted
[...]
const char *key;
const char *value;
+ int overwrite_on_reconfigure;
If you make this a "keep_on_reconfigure", then ...
I do not think that this would be a better name, or that renaming this
field would do anything except cause more work for me.
It would also result in more readable code, i.e. why add boilerplate ",
1" to a boolean field in this case if every single setting is set to
"1"? Doesn't it make more sense to invert the variable name & save on
the verbosity?
You won't need the churn/boilerplate of adding "1" to everything here,
but can just change the initial patch to use designated initializers.
That along with a throwaway macro like:
#define SCALAR_CFG_TRUE(k) (.key = k, .value = "true")
#define SCALAR_CFG_FALSE(k) (.key = k, .value = "false")
Might (or might not) make this even easier to eyeball...
To me, it makes things less readable. There is an entire section with the
header `/* Optional */` below, and I want this list to stay as readable as
it is now.
Yeah, I think those macros are probably less readable too. I should have
phrased that as a "one could even...", but just the smaller change of
avoiding the ", 1" everywhere seems worthwhile.
On Fri, Sep 03 2021, Johannes Schindelin via GitGitGadget wrote:
Changes since v1:
* A couple typos were fixed
* The code parsing the output of ls-remote was made more readable
* The indentation used in scalar.txt now consistently uses tabs
* We no longer hard-code core.bare = false when registering with Scalar
On Fri, Sep 03 2021, Johannes Schindelin via GitGitGadget wrote:
From: Johannes Schindelin <redacted>
This implements Scalar's opinionated `clone` command: it tries to use a
partial clone and sets up a sparse checkout by default. In contrast to
`git clone`, `scalar clone` sets up the worktree in the `src/`
subdirectory, to encourage a separation between the source files and the
build output (which helps Git tremendously because it avoids untracked
files that have to be specifically ignored when refreshing the index).
Perhaps it's simpler to just say about that /src/ injection:
`scalar clone` adds an implicit "/src" subdirectory to whatever
directory the user provides, with the added stricture on top of
doing that with "git clone" that the "src" cannot exist already.
...
+ if (is_directory(enlistment))
+ die(_("directory '%s' exists already"), enlistment);
+
+ dir = xstrfmt("%s/src", enlistment);
Which also seems to suggest a bug here. I.e. if I "git clone <repo>
/tmp/xyz/abc" and Ctrl+C it we'll remove "abc", but leave "xyz"
behind. Since we're creating that "xyz" (or "src") implicitly here an
abort/ctrl+C followed by a retry is going to run into this error, isn't
it?
I.e. it seems what's missing in this state machine is checking if the
directory was there already, and if it isn't add it to the existing
atexit() removals.
Which may be tricky seeing as this is shelling out to "init" then
"fetch" etc, i.e. who removes it? But maybe not.
From: Johannes Schindelin <hidden> Date: 2021-09-08 18:21:21
Hi Eric,
On Fri, 3 Sep 2021, Eric Sunshine wrote:
On Fri, Sep 3, 2021 at 11:23 AM Johannes Schindelin
[off-list ref] wrote:
quoted
On Tue, 31 Aug 2021, Eric Sunshine wrote:
quoted
On Mon, Aug 30, 2021 at 5:35 PM Johannes Schindelin via GitGitGadget
quoted
+With the `--all` option, all enlistments currently registered with Scalar
+will be reconfigured. This option is meant to to be run every time Scalar
+was upgraded.
s/was/is/
I wanted to convey a temporal order, so I changed it to "every time after
Scalar is upgraded". Okay?
I think I understood the intent of the original, but it causes a
grammatical hiccup. Your revised version can work, although I might
write it this way:
This option is meant to be run each time Scalar is upgraded.
However, perhaps that is too ambiguous and some users may think that
the process of upgrading Scalar will automatically run this command,
and you'd like to make it clear that it is the user's responsibility.
So, perhaps:
Use this option after each Scalar upgrade.
or something.
From: Johannes Schindelin <hidden> Date: 2021-09-08 19:11:55
Hi Bagas,
On Sat, 4 Sep 2021, Bagas Sanjaya wrote:
On 04/09/21 00.54, Derrick Stolee via GitGitGadget wrote:
quoted
+List
+~~~~
+
+list::
+ To see which repositories are currently registered by the service, run
+ `scalar list`. This subcommand does not need to be run inside a Scalar
+ enlistment.
+
I think the man-page-style wording should be:
quoted
list::
List enlistments that are currently registered by Scalar. This
subcommand does not need to be run inside an enlistment.
From: Johannes Schindelin <hidden> Date: 2021-09-08 19:23:29
Hi Ævar,
On Mon, 6 Sep 2021, Ævar Arnfjörð Bjarmason wrote:
On Fri, Sep 03 2021, Johannes Schindelin via GitGitGadget wrote:
quoted
From: Johannes Schindelin <redacted>
This implements Scalar's opinionated `clone` command: it tries to use a
partial clone and sets up a sparse checkout by default. In contrast to
`git clone`, `scalar clone` sets up the worktree in the `src/`
subdirectory, to encourage a separation between the source files and the
build output (which helps Git tremendously because it avoids untracked
files that have to be specifically ignored when refreshing the index).
Perhaps it's simpler to just say about that /src/ injection:
`scalar clone` adds an implicit "/src" subdirectory to whatever
directory the user provides, with the added stricture on top of
doing that with "git clone" that the "src" cannot exist already.
It would not only be simpler, it would also skip an important point I
tried to make, namely how this _differs_ from `git clone`. Rather crucial,
really.
...
quoted
+ if (is_directory(enlistment))
+ die(_("directory '%s' exists already"), enlistment);
+
+ dir = xstrfmt("%s/src", enlistment);
Which also seems to suggest a bug here. I.e. if I "git clone <repo>
/tmp/xyz/abc" and Ctrl+C it we'll remove "abc", but leave "xyz"
behind. Since we're creating that "xyz" (or "src") implicitly here an
abort/ctrl+C followed by a retry is going to run into this error, isn't
it?
Sure, it's just like calling `git clone <url> a/b/c` and upon failure
seeing only the `c` directory removed, while `a/b` is left behind.
I.e. it seems what's missing in this state machine is checking if the
directory was there already, and if it isn't add it to the existing
atexit() removals.
Which may be tricky seeing as this is shelling out to "init" then
"fetch" etc, i.e. who removes it? But maybe not.
I would rather spend time (after this patch series landed, of course) on
teaching `git clone` to handle what Scalar needs, and upon Ctrl+C to
optionally remove _all_ the directories it created, not just the innermost
one.
Then we get that functionality "for free", without spending a lot of time
on code that will be obsolete soon enough anyway.
Ciao,
Johannes
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-09-08 19:24:51
tl;dr: This series contributes the Scalar command to the Git project. This
command provides an opinionated way to create and configure repositories
with a focus on very large repositories.
Background
==========
Years ago, Microsoft wanted to move the source code of the Windows operating
system to Git. The challenge there was to prove that Git could scale to
massive monorepos. The VFS for Git (formerly GVFS) project was born to take
up that challenge.
The final solution included a virtual filesystem (with both user-mode and
kernel components) and a customized fork of Git for Windows. This solution
contained several key concepts, such as only populating a portion of the
working directory, demand-fetching blobs, and performing periodic repo
maintenance in the background. However, the required kernel drivers made it
difficult to port the solution to other platforms.
But it was realized that many of these key concepts were independent of the
actual VFS and its projection of the working directory. The Scalar project
was created to make that separation, refine the key concepts, and then
extract those features into the new Scalar command.
The present
===========
The Scalar project provides a completely functional non-virtual experience
for monorepos. But why stop there. The Scalar project was designed to be a
self-destructing vehicle to allow those key concepts to be moved into core
Git itself for the benefit of all. For example, partial clone,
sparse-checkout, and background maintenance have already been upstreamed and
removed from Scalar proper. This patch series provides a C-based
implementation of the final remaining portions of the Scalar command. This
will make it easier for users to experiment with the Scalar command. It will
also make it substantially easier to experiment with moving functionality
from Scalar into core Git, while maintaining backwards-compatibility for
existing Scalar users.
The C-based Scalar has been shipped to Scalar users, and can be tested by
any interested reader:
https://github.com/microsoft/git/releases/tag/v2.33.0.vfs.0.0 (it offers a
Git for Windows installer, a macOS package and an Ubuntu package).
Opportunities
=============
Apart from providing the Scalar command, this contribution is intended to
serve as a basis for further mailing list discussions on moving (some of)
these key concepts into the main Git commands.
For example, we previously discussed the idea of a "git big-clone" that does
much of what "scalar clone" is doing. This patch series is a step to make
such functionality exist in the Git code base while we simmer on what such a
"git big-clone" command-line interface would look like.
This is one of many possible ways to do this. Creating a 'git big-clone'
could lock Git into backwards compatibility concerns so it is necessary to
approach such an endeavor with caution. As a discussion starter, the scalar
clone <url> command does roughly this:
1. git clone --sparse --filter=blob:none /src
2. git -C /src sparse-checkout init --cone
3. git -C /src config (many times)
4. git -C /src maintenance start
It is my hope inspire discussions about what parts of Scalar could go into
core Git, and where, and in which form. While we wish to maintain
backwards-compatibility of Scalar's command-line interface (because it is
already in use), by having the Scalar code in the same code base as Git's,
it will be much easier to move functionality without having to maintain
loose version coupling between independently-versioned Scalar and Git. The
tight version-coupling, along with having access to libgit.a also allows the
C-based implementation of Scalar to be much smaller than the original .NET
version.
For example, we might choose in the future to implement, say, git clone
--scale=partial,cone to initialize a partial clone with a cone-sparse
checkout, that would not only be totally doable, and not only would we
already have precedent and data to prove that this actually makes engineers
happy who have to work on ginormous repositories, but we could then also
implement it by moving parts of contrib/scalar/ to builtin/ (where
contrib/scalar/ would then call the built-ins accordingly rather than
hard-coding the defaults itself).
We now also have the opportunity to discuss the merits of Scalar's clone
caching, which is not actually part of this patch series because it is a bit
coupled with the GVFS parts of microsoft/git for the moment, where clones
automatically get registered with a populated alternate repository that is
identified by the URL, meaning: subsequent clones of the same repository are
vastly faster than the first one because they do not actually download the
already-received objects again, they access the cache instead.
Another thing that I could imagine to be discussed at length is the
distinction between enlistment and worktree (where the latter is the actual
Git worktree and usually lives in the src/ subdirectory of the former). This
encourages untracked and ignored files to be placed outside the worktree,
making Git's job much easier. This idea, too, might find its way in one way
or another into Git proper.
These are just a few concepts in Scalar that do not yet have equivalents in
Git. By putting this initial implementation into contrib/, we create a
foundation for future discussions of these concepts.
We plan on updating the recommended config settings in scalar register as
new Git features are available (such as builtin FSMonitor and sparse-index,
when ready). To facilitate upgrading existing Scalar enlistments, their
paths are automatically added to the [scalar] section of the global Git
config, and the scalar reconfigure --all command will process all of them.
Epilogue
========
Now, to address some questions that I imagine every reader has who made it
this far:
* Why not put the Scalar functionality directly into a built-in? Creating a
Git builtin requires scrutiny over every aspect of the feature, which is
difficult to do while also maintaining the command-line interface
contract and expected behavior of the Scalar command (there are existing
users, after all). By having the Scalar command in contrib/, we present a
simple option for users to have these features in the short term while
the Git contributor community decides which bits to absorb into Git
built-ins.
* Why implement the Scalar command in the Git codebase? We ported Scalar to
the microsoft/git fork for several reasons. First, we realized it was
possible now that the core features exist inside Git itself. Second,
compiling Scalar directly within a version of Git allows us to remove a
version compatibility check from each config option that might or might
not apply based on the installed Git version. Finally, this new location
has greatly simplified our release process and the installation process
for users. We now have ways to install Scalar with microsoft/git via
winget, brew, and apt-get. This has been the case since we shipped
v2.32.0 to our users, read: this setup has served us well already.
* Why contribute Scalar to the Git project? We are biased, of course, yet
we do have evidence that the Scalar command is a helpful tool that offers
an simple way to handle huge repositories with ease. By contributing it
to the core Git project, we are able to share it with more users,
especially some users who do not want to install the microsoft/git fork.
We intend to include Scalar as a component in git-for-windows/git, but
are contributing it here first. Further, we think there is benefit to the
Git developer community as this presents an example of how to set certain
defaults that work for large repositories.
* Does this integrate with the built-in FSMonitor yet? No, not yet. I do
have a couple of add-on patch series lined up, one of them being the
integration with the built-in FSMonitor, which obviously has to wait
until the FSMonitor patch series advances further.
Changes since v2:
* Adjusted the description of the list command in the manual page , as
suggested by Bagas.
* Addressed two style nits in cmd_run().
* The documentation of git reconfigure -a was improved.
Changes since v1:
* A couple typos were fixed
* The code parsing the output of ls-remote was made more readable
* The indentation used in scalar.txt now consistently uses tabs
* We no longer hard-code core.bare = false when registering with Scalar
Derrick Stolee (4):
scalar: 'register' sets recommended config and starts maintenance
scalar: 'unregister' stops background maintenance
scalar: implement 'scalar list'
scalar: implement the `run` command
Johannes Schindelin (10):
scalar: create a rudimentary executable
scalar: start documenting the command
scalar: create test infrastructure
scalar: let 'unregister' handle a deleted enlistment directory
gracefully
scalar: implement the `clone` subcommand
scalar: teach 'clone' to support the --single-branch option
scalar: allow reconfiguring an existing enlistment
scalar: teach 'reconfigure' to optionally handle all registered
enlistments
scalar: implement the `version` command
scalar: accept -C and -c options before the subcommand
Matthew John Cheetham (1):
scalar: implement the `delete` command
Makefile | 8 +
contrib/scalar/.gitignore | 5 +
contrib/scalar/Makefile | 57 +++
contrib/scalar/scalar.c | 844 +++++++++++++++++++++++++++++++
contrib/scalar/scalar.txt | 154 ++++++
contrib/scalar/t/Makefile | 78 +++
contrib/scalar/t/t9099-scalar.sh | 88 ++++
7 files changed, 1234 insertions(+)
create mode 100644 contrib/scalar/.gitignore
create mode 100644 contrib/scalar/Makefile
create mode 100644 contrib/scalar/scalar.c
create mode 100644 contrib/scalar/scalar.txt
create mode 100644 contrib/scalar/t/Makefile
create mode 100755 contrib/scalar/t/t9099-scalar.sh
base-commit: ebf3c04b262aa27fbb97f8a0156c2347fecafafb
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1005%2Fdscho%2Fscalar-the-beginning-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1005/dscho/scalar-the-beginning-v3
Pull-Request: https://github.com/gitgitgadget/git/pull/1005
Range-diff vs v2:
1: b8c7d3f8450 = 1: b8c7d3f8450 scalar: create a rudimentary executable
2: 4f886575dcf = 2: 4f886575dcf scalar: start documenting the command
3: bcfde9bc765 = 3: bcfde9bc765 scalar: create test infrastructure
4: ee3e26a0c4e = 4: ee3e26a0c4e scalar: 'register' sets recommended config and starts maintenance
5: 6142f75875b = 5: 6142f75875b scalar: 'unregister' stops background maintenance
6: 82dd253154f = 6: 82dd253154f scalar: let 'unregister' handle a deleted enlistment directory gracefully
7: fb7c931ddb3 ! 7: d291d3723a6 scalar: implement 'scalar list'
@@ contrib/scalar/scalar.txt: an existing Git worktree with Scalar whose name is no
+~~~~
+
+list::
-+ To see which repositories are currently registered by the service, run
-+ `scalar list`. This subcommand does not need to be run inside a Scalar
-+ enlistment.
++ List enlistments that are currently registered by Scalar. This
++ subcommand does not need to be run inside an enlistment.
+
Register
~~~~~~~~
8: f3223c10788 ! 8: 40dbf61771e scalar: implement the `clone` subcommand
@@ contrib/scalar/scalar.txt: an existing Git worktree with Scalar whose name is no
List
~~~~
- list::
- To see which repositories are currently registered by the service, run
-- `scalar list`. This subcommand does not need to be run inside a Scalar
-- enlistment.
-+ `scalar list`. This subcommand, like `clone`, does not need to be run
-+ inside a Scalar enlistment.
-
- Register
- ~~~~~~~~
@@ contrib/scalar/scalar.txt: unregister [<enlistment>]::
SEE ALSO
9: b3c4b3dccc6 = 9: 414dbe7d859 scalar: teach 'clone' to support the --single-branch option
10: b7fc2dc29c8 ! 10: 76de416a643 scalar: implement the `run` command
@@ contrib/scalar/scalar.c: static int cmd_register(int argc, const char **argv)
+ argc = parse_options(argc, argv, NULL, options,
+ usagestr, 0);
+
-+ if (argc == 0)
++ if (!argc)
+ usage_with_options(usagestr, options);
+
-+ if (!strcmp("all", argv[0]))
++ if (!strcmp("all", argv[0])) {
+ i = -1;
-+ else {
++ } else {
+ for (i = 0; tasks[i].arg && strcmp(tasks[i].arg, argv[0]); i++)
+ ; /* keep looking for the task */
+
11: 9a834c23d08 = 11: 655a902b9df scalar: allow reconfiguring an existing enlistment
12: 79e9f5d203a ! 12: 2d1987bfcda scalar: teach 'reconfigure' to optionally handle all registered enlistments
@@ contrib/scalar/scalar.txt: After a Scalar upgrade, or when the configuration of
reconfigure the enlistment.
+With the `--all` option, all enlistments currently registered with Scalar
-+will be reconfigured. This option is meant to to be run every time after
-+Scalar is upgraded.
++will be reconfigured. Use this option after each Scalar upgrade.
+
SEE ALSO
--------
13: 94a21982652 ! 13: c67938299ee scalar: implement the `delete` command
@@ contrib/scalar/scalar.txt: scalar register [<enlistment>]
DESCRIPTION
-----------
-@@ contrib/scalar/scalar.txt: With the `--all` option, all enlistments currently registered with Scalar
- will be reconfigured. This option is meant to to be run every time after
- Scalar is upgraded.
+@@ contrib/scalar/scalar.txt: reconfigure the enlistment.
+ With the `--all` option, all enlistments currently registered with Scalar
+ will be reconfigured. Use this option after each Scalar upgrade.
+Delete
+~~~~~~
14: 707d8e19683 = 14: d2cd2b7094b scalar: implement the `version` command
15: 26e23b5c5e5 = 15: 7ccc4f8b9b0 scalar: accept -C and -c options before the subcommand
--
gitgitgadget
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-09-08 19:24:51
From: Johannes Schindelin <redacted>
The idea of Scalar (https://github.com/microsoft/scalar), and before
that, of VFS for Git, has always been to prove that Git _can_ scale, and
to upstream whatever strategies have been demonstrated to help.
With this patch, we start the journey from that C# project to move what
is left to Git's own `contrib/` directory, reimplementing it in pure C,
with the intention to facilitate integrating the functionality into core
Git all while maintaining backwards-compatibility for existing Scalar
users (which will be much easier when both live in the same worktree).
It was always to plan to contribute all of the proven strategies back to
core Git.
For example, while the virtual filesystem provided by VFS for Git helped
the team developing the Windows operating system to move onto Git, while
trying to upstream it we realized that it cannot be done: getting the
virtual filesystem to work (which we only managed to implement fully on
Windows, but not on, say, macOS or Linux), and the required server-side
support for the GVFS protocol, made this not quite feasible.
The Scalar project learned from that and tackled the problem with
different tactics: instead of pretending to Git that the working
directory is fully populated, it _specifically_ teaches Git about
partial clone (which is based on VFS for Git's cache server), about
sparse checkout (which VFS for Git tried to do transparently, in the
file system layer), and regularly runs maintenance tasks to keep the
repository in a healthy state.
With partial clone, sparse checkout and `git maintenance` having been
upstreamed, there is little left that `scalar.exe` does that which
`git.exe` cannot do. One such thing is that `scalar clone <url>` will
automatically set up a partial, sparse clone, and configure
known-helpful settings from the start.
So let's bring this convenience into Git's tree.
The idea here is that you can (optionally) build Scalar via
make -C contrib/scalar/Makefile
This will build the `scalar` executable and put it into the
contrib/scalar/ subdirectory.
The slightly awkward addition of the `contrib/scalar/*` bits to the
top-level `Makefile` are actually really required: we want to link to
`libgit.a`, which means that we will need to use the very same `CFLAGS`
and `LDFLAGS` as the rest of Git.
An early development version of this patch tried to replicate all the
conditional code in `contrib/scalar/Makefile` (e.g. `NO_POLL`) just like
`contrib/svn-fe/Makefile` used to do before it was retired. It turned
out to be quite the whack-a-mole game: the SHA-1-related flags, the
flags enabling/disabling `compat/poll/`, `compat/regex/`,
`compat/win32mmap.c` & friends depending on the current platform... To
put it mildly: it was a major mess.
Instead, this patch makes minimal changes to the top-level `Makefile` so
that the bits in `contrib/scalar/` can be compiled and linked, and
adds a `contrib/scalar/Makefile` that uses the top-level `Makefile` in a
most minimal way to do the actual compiling.
Note: With this commit, we only establish the infrastructure, no
Scalar functionality is implemented yet; We will do that incrementally
over the next few commits.
Signed-off-by: Johannes Schindelin <redacted>
---
Makefile | 8 ++++++++
contrib/scalar/.gitignore | 2 ++
contrib/scalar/Makefile | 34 ++++++++++++++++++++++++++++++++++
contrib/scalar/scalar.c | 36 ++++++++++++++++++++++++++++++++++++
4 files changed, 80 insertions(+)
create mode 100644 contrib/scalar/.gitignore
create mode 100644 contrib/scalar/Makefile
create mode 100644 contrib/scalar/scalar.c
@@ -0,0 +1,34 @@+QUIET_SUBDIR0=+$(MAKE)-C# space to separate -C and subdir+QUIET_SUBDIR1=++ifneq ($(findstring s,$(MAKEFLAGS)),s)+ifndef V+QUIET_SUBDIR0=+@subdir=+QUIET_SUBDIR1=;$(NO_SUBDIR)echo' 'SUBDIR$$subdir;\+$(MAKE)$(PRINT_DIR)-C$$subdir+else+exportV+endif+endif++all:++include ../../config.mak.uname+-include ../../config.mak.autogen+-include ../../config.mak++TARGETS=scalar$(X)scalar.o+GITLIBS=../../common-main.o../../libgit.a../../xdiff/lib.a++all:scalar$X++$(GITLIBS):+$(QUIET_SUBDIR0)../..$(QUIET_SUBDIR1)$(subst../../,,$@)++$(TARGETS):$(GITLIBS)scalar.c+$(QUIET_SUBDIR0)../..$(QUIET_SUBDIR1)$(patsubst%,contrib/scalar/%,$@)++clean:+$(RM)$(TARGETS)++.PHONY:allcleanFORCE
@@ -30,5 +31,16 @@ $(TARGETS): $(GITLIBS) scalar.cclean:$(RM)$(TARGETS)+$(RM)scalar.1scalar.htmlscalar.xml-.PHONY:allcleanFORCE+docs:scalar.htmlscalar.1++scalar.html:|scalar.1 # prevent them from trying to build `doc.dep` in parallel++scalar.html scalar.1:scalar.txt+$(QUIET_SUBDIR0)../../Documentation$(QUIET_SUBDIR1)\+MAN_TXT=../contrib/scalar/scalar.txt\+../contrib/scalar/$@+$(QUIET)testscalar.1!="$@"||mv../../Documentation/$@.++.PHONY:allcleandocsFORCE
@@ -0,0 +1,38 @@+scalar(1)+=========++NAME+----+scalar - an opinionated repository management tool++SYNOPSIS+--------+[verse]+scalar <command> [<options>]++DESCRIPTION+-----------++Scalar is an opinionated repository management tool. By creating new+repositories or registering existing repositories with Scalar, your Git+experience will speed up. Scalar sets advanced Git config settings,+maintains your repositories in the background, and helps reduce data sent+across the network.++An important Scalar concept is the enlistment: this is the top-level directory+of the project. It usually contains the subdirectory `src/` which is a Git+worktree. This encourages the separation between tracked files (inside `src/`)+and untracked files, such as build artifacts (outside `src/`). When registering+an existing Git worktree with Scalar whose name is not `src`, the enlistment+will be identical to the worktree.++The `scalar` command implements various subcommands, and different options+depending on the subcommand.++SEE ALSO+--------+linkgit:git-maintenance[1].++Scalar+---+Associated with the linkgit:git[1] suite
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-09-08 19:24:57
From: Johannes Schindelin <redacted>
To test the Scalar command, create a test script in contrib/scalar/t
that is executed as `make -C contrib/scalar test`. Since Scalar has no
meaningful capabilities yet, the only test is rather simple. We will add
more tests in subsequent commits that introduce corresponding, new
functionality.
Note: this test script is intended to test `scalar` only lightly, even
after all of the functionality is implemented.
A more comprehensive functional (or: integration) test suite can be
found at https://github.com/microsoft/scalar; It is used in the workflow
https://github.com/microsoft/git/blob/HEAD/.github/workflows/scalar-functional-tests.yml
in Microsoft's Git fork. This test suite performs end-to-end tests with
a real remote repository, and is run as part of the regular CI builds.
Since those tests require some functionality supported only by
Microsoft's Git fork ("GVFS protocol"), there is no intention to port
that fuller test suite to `contrib/scalar/`.
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/Makefile | 17 +++++--
contrib/scalar/t/Makefile | 78 ++++++++++++++++++++++++++++++++
contrib/scalar/t/t9099-scalar.sh | 17 +++++++
3 files changed, 109 insertions(+), 3 deletions(-)
create mode 100644 contrib/scalar/t/Makefile
create mode 100755 contrib/scalar/t/t9099-scalar.sh
@@ -21,7 +22,7 @@ include ../../config.mak.unameTARGETS=scalar$(X)scalar.oGITLIBS=../../common-main.o../../libgit.a../../xdiff/lib.a-all:scalar$X+all:scalar$X ../../bin-wrappers/scalar$(GITLIBS):$(QUIET_SUBDIR0)../..$(QUIET_SUBDIR1)$(subst../../,,$@)
@@ -30,9 +31,19 @@ $(TARGETS): $(GITLIBS) scalar.c$(QUIET_SUBDIR0)../..$(QUIET_SUBDIR1)$(patsubst%,contrib/scalar/%,$@)clean:-$(RM)$(TARGETS)+$(RM)$(TARGETS)../../bin-wrappers/scalar$(RM)scalar.1scalar.htmlscalar.xml+../../bin-wrappers/scalar:../../wrap-for-bin.shMakefile+@mkdir-p../../bin-wrappers+$(QUIET_GEN)sed-e'1s|#!.*/sh|#!$(SHELL_PATH_SQ)|'\+-e's|@@BUILD_DIR@@|$(shell cd ../.. && pwd)|'\+-e's|@@PROG@@|contrib/scalar/scalar$(X)|'<$<>$@&&\+chmod+x$@++test:all+$(MAKE)-Ct+docs:scalar.htmlscalar.1scalar.html:|scalar.1 # prevent them from trying to build `doc.dep` in parallel
@@ -0,0 +1,17 @@+#!/bin/sh++test_description='test the `scalar` command'++TEST_DIRECTORY=$PWD/../../../t+exportTEST_DIRECTORY++# Make it work with --no-bin-wrappers+PATH=$PWD/..:$PATH++.../../../t/test-lib.sh++test_expect_success'scalar shows a usage''+test_expect_code129scalar-h+'++test_done
From: Derrick Stolee via GitGitGadget <hidden> Date: 2021-09-08 19:24:57
From: Derrick Stolee <redacted>
Let's start implementing the `register` command. With this commit,
recommended settings are configured upon `scalar register`, and Git's
background maintenance is started.
The recommended config settings may very well change in the future. For
example, once the built-in FSMonitor is available, we will want to
enable it upon `scalar register`. For that reason, we explicitly support
running `scalar register` in an already-registered enlistment.
Co-authored-by: Victoria Dye [off-list ref]
Signed-off-by: Derrick Stolee <redacted>
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 255 ++++++++++++++++++++++++++++++++++++++
contrib/scalar/scalar.txt | 18 ++-
2 files changed, 272 insertions(+), 1 deletion(-)
@@ -5,11 +5,266 @@#include"cache.h"#include"gettext.h"#include"parse-options.h"+#include"config.h"+#include"run-command.h"++/*+*Removethedeepestsubdirectoryintheprovidedpathstring.Pathmustnot+*includeatrailingpathseparator.Returns1ifparentdirectoryfound,+*otherwise0.+*/+staticintstrbuf_parent_directory(structstrbuf*buf)+{+size_tlen=buf->len;+size_toffset=offset_1st_component(buf->buf);+char*path_sep=find_last_dir_sep(buf->buf+offset);+strbuf_setlen(buf,path_sep?path_sep-buf->buf:offset);++returnbuf->len<len;+}++staticvoidsetup_enlistment_directory(intargc,constchar**argv,+constchar*const*usagestr,+conststructoption*options,+structstrbuf*enlistment_root)+{+structstrbufpath=STRBUF_INIT;+char*root;+intenlistment_found=0;++if(startup_info->have_repository)+BUG("gitdir already set up?!?");++if(argc>1)+usage_with_options(usagestr,options);++/* find the worktree, determine its corresponding root */+if(argc==1)+strbuf_add_absolute_path(&path,argv[0]);+elseif(strbuf_getcwd(&path)<0)+die(_("need a working directory"));++strbuf_trim_trailing_dir_sep(&path);+do{+constsize_tlen=path.len;++/* check if currently in enlistment root with src/ workdir */+strbuf_addstr(&path,"/src/.git");+if(is_git_directory(path.buf)){+strbuf_strip_suffix(&path,"/.git");++if(enlistment_root)+strbuf_add(enlistment_root,path.buf,len);++enlistment_found=1;+break;+}++/* reset to original path */+strbuf_setlen(&path,len);++/* check if currently in workdir */+strbuf_addstr(&path,"/.git");+if(is_git_directory(path.buf)){+strbuf_setlen(&path,len);++if(enlistment_root){+/*+*Iftheworktree'sdirectory'snameis`src`,theenlistmentisthe+*parentdirectory,otherwiseitisidenticaltotheworktree.+*/+root=strip_path_suffix(path.buf,"src");+strbuf_addstr(enlistment_root,root?root:path.buf);+free(root);+}++enlistment_found=1;+break;+}++strbuf_setlen(&path,len);+}while(strbuf_parent_directory(&path));++if(!enlistment_found)+die(_("could not find enlistment root"));++if(chdir(path.buf)<0)+die_errno(_("could not switch to '%s'"),path.buf);++strbuf_release(&path);+setup_git_directory();+}++staticintrun_git(constchar*arg,...)+{+structstrvecargv=STRVEC_INIT;+va_listargs;+constchar*p;+intres;++va_start(args,arg);+strvec_push(&argv,arg);+while((p=va_arg(args,constchar*)))+strvec_push(&argv,p);+va_end(args);++res=run_command_v_opt(argv.v,RUN_GIT_CMD);++strvec_clear(&argv);+returnres;+}++staticintset_recommended_config(void)+{+struct{+constchar*key;+constchar*value;+}config[]={+{"am.keepCR","true"},+{"core.FSCache","true"},+{"core.multiPackIndex","true"},+{"core.preloadIndex","true"},+#ifndef WIN32+{"core.untrackedCache","true"},+#else+/*+*Unfortunately,Scalar'sFunctionalTestsdemonstrated+*thattheuntrackedcachefeatureisunreliableonWindows+*(whichisabummerbecausethatplatformwouldbenefitthe+*mostfromit).Forsomereason,freshlycreatedfilesseem+*nottoupdatethedirectory's`lastModified`time+*immediately,buttheuntrackedcachewouldneedtorelyon+*that.+*+*Therefore,withasadheart,wedisablethisveryuseful+*featureonWindows.+*/+{"core.untrackedCache","false"},+#endif+{"core.logAllRefUpdates","true"},+{"credential.https://dev.azure.com.useHttpPath","true"},+{"credential.validate","false"},/* GCM4W-only */+{"gc.auto","0"},+{"gui.GCWarning","false"},+{"index.threads","true"},+{"index.version","4"},+{"merge.stat","false"},+{"merge.renames","false"},+{"pack.useBitmaps","false"},+{"pack.useSparse","true"},+{"receive.autoGC","false"},+{"reset.quiet","true"},+{"feature.manyFiles","false"},+{"feature.experimental","false"},+{"fetch.unpackLimit","1"},+{"fetch.writeCommitGraph","false"},+#ifdef WIN32+{"http.sslBackend","schannel"},+#endif+{"status.aheadBehind","false"},+{"commitGraph.generationVersion","1"},+{"core.autoCRLF","false"},+{"core.safeCRLF","false"},+{NULL,NULL},+};+inti;+char*value;++for(i=0;config[i].key;i++){+if(git_config_get_string(config[i].key,&value)){+trace2_data_string("scalar",the_repository,config[i].key,"created");+if(git_config_set_gently(config[i].key,+config[i].value)<0)+returnerror(_("could not configure %s=%s"),+config[i].key,config[i].value);+}else{+trace2_data_string("scalar",the_repository,config[i].key,"exists");+free(value);+}+}++/*+*The`log.excludeDecoration`settingisspecialbecauseitallows+*formultiplevalues.+*/+if(git_config_get_string("log.excludeDecoration",&value)){+trace2_data_string("scalar",the_repository,+"log.excludeDecoration","created");+if(git_config_set_multivar_gently("log.excludeDecoration",+"refs/prefetch/*",+CONFIG_REGEX_NONE,0))+returnerror(_("could not configure "+"log.excludeDecoration"));+}else{+trace2_data_string("scalar",the_repository,+"log.excludeDecoration","exists");+free(value);+}++return0;+}++staticintstart_maintenance(void)+{+returnrun_git("maintenance","start",NULL);+}++staticintadd_enlistment(void)+{+intres;++if(!the_repository->worktree)+die(_("Scalar enlistments require a worktree"));++res=run_git("config","--global","--get","--fixed-value",+"scalar.repo",the_repository->worktree,NULL);++/*+*Ifthesettingisalreadythere,thendonothing.+*/+if(!res)+return0;++returnrun_git("config","--global","--add",+"scalar.repo",the_repository->worktree,NULL);+}++staticintregister_dir(void)+{+intres=add_enlistment();++if(!res)+res=set_recommended_config();++if(!res)+res=start_maintenance();++returnres;+}++staticintcmd_register(intargc,constchar**argv)+{+structoptionoptions[]={+OPT_END(),+};+constchar*constusage[]={+N_("scalar register [<enlistment>]"),+NULL+};++argc=parse_options(argc,argv,NULL,options,+usage,0);++setup_enlistment_directory(argc,argv,usage,options,NULL);++returnregister_dir();+}staticstruct{constchar*name;int(*fn)(int,constchar**);}builtins[]={+{"register",cmd_register},{NULL,NULL},};
@@ -29,6 +29,22 @@ will be identical to the worktree. The `scalar` command implements various subcommands, and different options depending on the subcommand.+COMMANDS+--------++Register+~~~~~~~~++register [<enlistment>]::+ Adds the enlistment's repository to the list of registered repositories+ and starts background maintenance. If `<enlistment>` is not provided,+ then the enlistment associated with the current working directory is+ registered.+++Note: when this subcommand is called in a worktree that is called `src/`, its+parent directory is considered to be the Scalar enlistment. If the worktree is+_not_ called `src/`, it itself will be considered to be the Scalar enlistment.+ SEE ALSO -------- linkgit:git-maintenance[1].
From: Derrick Stolee via GitGitGadget <hidden> Date: 2021-09-08 19:25:00
From: Derrick Stolee <redacted>
Just like `scalar register` starts the scheduled background maintenance,
`scalar unregister` stops it. Note that we use `git maintenance start`
in `scalar register`, but we do not use `git maintenance stop` in
`scalar unregister`: this would stop maintenance for _all_ repositories,
not just for the one we want to unregister.
The `unregister` command also removes the corresponding entry from the
`[scalar]` section in the global Git config.
Co-authored-by: Victoria Dye [off-list ref]
Signed-off-by: Derrick Stolee <redacted>
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 50 ++++++++++++++++++++++++++++++++-------
contrib/scalar/scalar.txt | 8 +++++++
2 files changed, 50 insertions(+), 8 deletions(-)
@@ -45,6 +46,13 @@ Note: when this subcommand is called in a worktree that is called `src/`, its parent directory is considered to be the Scalar enlistment. If the worktree is _not_ called `src/`, it itself will be considered to be the Scalar enlistment.+Unregister+~~~~~~~~~~++unregister [<enlistment>]::+ Remove the specified repository from the list of repositories+ registered with Scalar and stop the scheduled background maintenance.+ SEE ALSO -------- linkgit:git-maintenance[1].
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-09-08 19:25:04
From: Johannes Schindelin <redacted>
When a user deleted an enlistment manually, let's be generous and
_still_ unregister it.
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 46 ++++++++++++++++++++++++++++++++
contrib/scalar/t/t9099-scalar.sh | 15 +++++++++++
2 files changed, 61 insertions(+)
From: Derrick Stolee via GitGitGadget <hidden> Date: 2021-09-08 19:25:05
From: Derrick Stolee <redacted>
The produced list simply consists of those repositories registered under
the multi-valued `scalar.repo` config setting in the user's Git config.
Signed-off-by: Derrick Stolee <redacted>
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 11 +++++++++++
contrib/scalar/scalar.txt | 11 ++++++++++-
2 files changed, 21 insertions(+), 1 deletion(-)
@@ -257,6 +257,16 @@ static int unregister_dir(void)returnres;}+staticintcmd_list(intargc,constchar**argv)+{+if(argc!=1)+die(_("`scalar list` does not take arguments"));++if(run_git("config","--global","--get-all","scalar.repo",NULL)<0)+return-1;+return0;+}+staticintcmd_register(intargc,constchar**argv){structoptionoptions[]={
@@ -28,11 +29,19 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment will be identical to the worktree. The `scalar` command implements various subcommands, and different options-depending on the subcommand.+depending on the subcommand. With the exception of `list`, all subcommands+expect to be run in an enlistment. COMMANDS --------+List+~~~~++list::+ List enlistments that are currently registered by Scalar. This+ subcommand does not need to be run inside an enlistment.+ Register ~~~~~~~~
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-09-08 19:25:08
From: Johannes Schindelin <redacted>
This implements Scalar's opinionated `clone` command: it tries to use a
partial clone and sets up a sparse checkout by default. In contrast to
`git clone`, `scalar clone` sets up the worktree in the `src/`
subdirectory, to encourage a separation between the source files and the
build output (which helps Git tremendously because it avoids untracked
files that have to be specifically ignored when refreshing the index).
Also, it registers the repository for regular, scheduled maintenance,
and configures a flurry of configuration settings based on the
experience and experiments of the Microsoft Windows and the Microsoft
Office development teams.
Note: since the `scalar clone` command is by far the most commonly
called `scalar` subcommand, we document it at the top of the manual
page.
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 201 +++++++++++++++++++++++++++++++
contrib/scalar/scalar.txt | 31 ++++-
contrib/scalar/t/t9099-scalar.sh | 32 +++++
3 files changed, 261 insertions(+), 3 deletions(-)
@@ -257,6 +258,205 @@ static int unregister_dir(void)returnres;}+/* printf-style interface, expects `<key>=<value>` argument */+staticintset_config(constchar*fmt,...)+{+structstrbufbuf=STRBUF_INIT;+char*value;+intres;+va_listargs;++va_start(args,fmt);+strbuf_vaddf(&buf,fmt,args);+va_end(args);++value=strchr(buf.buf,'=');+if(value)+*(value++)='\0';+res=git_config_set_gently(buf.buf,value);+strbuf_release(&buf);++returnres;+}++staticchar*remote_default_branch(constchar*url)+{+structchild_processcp=CHILD_PROCESS_INIT;+structstrbufout=STRBUF_INIT;++cp.git_cmd=1;+strvec_pushl(&cp.args,"ls-remote","--symref",url,"HEAD",NULL);+if(!pipe_command(&cp,NULL,0,&out,0,NULL,0)){+constchar*line=out.buf;++while(*line){+constchar*eol=strchrnul(line,'\n'),*p;+size_tlen=eol-line;+char*branch;++if(!skip_prefix(line,"ref: ",&p)||+!strip_suffix_mem(line,&len,"\tHEAD")){+line=eol+(*eol=='\n');+continue;+}++eol=line+len;+if(skip_prefix(p,"refs/heads/",&p)){+branch=xstrndup(p,eol-p);+strbuf_release(&out);+returnbranch;+}++error(_("remote HEAD is not a branch: '%.*s'"),+(int)(eol-p),p);+strbuf_release(&out);+returnNULL;+}+}+warning(_("failed to get default branch name from remote; "+"using local default"));+strbuf_reset(&out);++child_process_init(&cp);+cp.git_cmd=1;+strvec_pushl(&cp.args,"symbolic-ref","--short","HEAD",NULL);+if(!pipe_command(&cp,NULL,0,&out,0,NULL,0)){+strbuf_trim(&out);+returnstrbuf_detach(&out,NULL);+}++strbuf_release(&out);+error(_("failed to get default branch name"));+returnNULL;+}++staticintcmd_clone(intargc,constchar**argv)+{+constchar*branch=NULL;+intfull_clone=0;+structoptionclone_options[]={+OPT_STRING('b',"branch",&branch,N_("<branch>"),+N_("branch to checkout after clone")),+OPT_BOOL(0,"full-clone",&full_clone,+N_("when cloning, create full working directory")),+OPT_END(),+};+constchar*constclone_usage[]={+N_("scalar clone [<options>] [--] <repo> [<dir>]"),+NULL+};+constchar*url;+char*enlistment=NULL,*dir=NULL;+structstrbufbuf=STRBUF_INIT;+intres;++argc=parse_options(argc,argv,NULL,clone_options,clone_usage,0);++if(argc==2){+url=argv[0];+enlistment=xstrdup(argv[1]);+}elseif(argc==1){+url=argv[0];++strbuf_addstr(&buf,url);+/* Strip trailing slashes, if any */+while(buf.len>0&&is_dir_sep(buf.buf[buf.len-1]))+strbuf_setlen(&buf,buf.len-1);+/* Strip suffix `.git`, if any */+strbuf_strip_suffix(&buf,".git");++enlistment=find_last_dir_sep(buf.buf);+if(!enlistment){+die(_("cannot deduce worktree name from '%s'"),url);+}+enlistment=xstrdup(enlistment+1);+}else{+usage_msg_opt(_("You must specify a repository to clone."),+clone_usage,clone_options);+}++if(is_directory(enlistment))+die(_("directory '%s' exists already"),enlistment);++dir=xstrfmt("%s/src",enlistment);++strbuf_reset(&buf);+if(branch)+strbuf_addf(&buf,"init.defaultBranch=%s",branch);+else{+char*b=repo_default_branch_name(the_repository,1);+strbuf_addf(&buf,"init.defaultBranch=%s",b);+free(b);+}++if((res=run_git("-c",buf.buf,"init","--",dir,NULL)))+gotocleanup;++if(chdir(dir)<0){+res=error_errno(_("could not switch to '%s'"),dir);+gotocleanup;+}++setup_git_directory();++/* common-main already logs `argv` */+trace2_def_repo(the_repository);++if(!branch&&!(branch=remote_default_branch(url))){+res=error(_("failed to get default branch for '%s'"),url);+gotocleanup;+}++if(set_config("remote.origin.url=%s",url)||+set_config("remote.origin.fetch="+"+refs/heads/*:refs/remotes/origin/*")||+set_config("remote.origin.promisor=true")||+set_config("remote.origin.partialCloneFilter=blob:none")){+res=error(_("could not configure remote in '%s'"),dir);+gotocleanup;+}++if(!full_clone&&+(res=run_git("sparse-checkout","init","--cone",NULL)))+gotocleanup;++if(set_recommended_config())+returnerror(_("could not configure '%s'"),dir);++if((res=run_git("fetch","--quiet","origin",NULL))){+warning(_("partial clone failed; attempting full clone"));++if(set_config("remote.origin.promisor")||+set_config("remote.origin.partialCloneFilter")){+res=error(_("could not configure for full clone"));+gotocleanup;+}++if((res=run_git("fetch","--quiet","origin",NULL)))+gotocleanup;+}++if((res=set_config("branch.%s.remote=origin",branch)))+gotocleanup;+if((res=set_config("branch.%s.merge=refs/heads/%s",+branch,branch)))+gotocleanup;++strbuf_reset(&buf);+strbuf_addf(&buf,"origin/%s",branch);+res=run_git("checkout","-f","-t",buf.buf,NULL);+if(res)+gotocleanup;++res=register_dir();++cleanup:+free(enlistment);+free(dir);+strbuf_release(&buf);+returnres;+}+staticintcmd_list(intargc,constchar**argv){if(argc!=1)
@@ -29,12 +30,36 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment will be identical to the worktree. The `scalar` command implements various subcommands, and different options-depending on the subcommand. With the exception of `list`, all subcommands-expect to be run in an enlistment.+depending on the subcommand. With the exception of `clone` and `list`, all+subcommands expect to be run in an enlistment. COMMANDS --------+Clone+~~~~~++clone [<options>] <url> [<enlistment>]::+ Clones the specified repository, similar to linkgit:git-clone[1]. By+ default, only commit and tree objects are cloned. Once finished, the+ worktree is located at `<enlistment>/src`.+++The sparse-checkout feature is enabled (except when run with `--full-clone`)+and the only files present are those in the top-level directory. Use+`git sparse-checkout set` to expand the set of directories you want to see,+or `git sparse-checkout disable` to expand to all files (see+linkgit:git-sparse-checkout[1] for more details). You can explore the+subdirectories outside your sparse-checkout by using `git ls-tree HEAD`.++-b <name>::+--branch <name>::+ Instead of checking out the branch pointed to by the cloned+ repository's HEAD, check out the `<name>` branch instead.++--[no-]full-clone::+ A sparse-checkout is initialized by default. This behavior can be+ turned off via `--full-clone`.+ List ~~~~
@@ -64,7 +89,7 @@ unregister [<enlistment>]:: SEE ALSO ---------linkgit:git-maintenance[1].+linkgit:git-clone[1], linkgit:git-maintenance[1]. Scalar ---
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-09-08 19:25:09
From: Johannes Schindelin <redacted>
Just like `git clone`, the `scalar clone` command now also offers to
restrict the clone to a single branch.
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 9 +++++++--
contrib/scalar/scalar.txt | 12 +++++++++++-
contrib/scalar/t/t9099-scalar.sh | 6 +++++-
3 files changed, 23 insertions(+), 4 deletions(-)
@@ -333,12 +333,15 @@ static char *remote_default_branch(const char *url)staticintcmd_clone(intargc,constchar**argv){constchar*branch=NULL;-intfull_clone=0;+intfull_clone=0,single_branch=0;structoptionclone_options[]={OPT_STRING('b',"branch",&branch,N_("<branch>"),N_("branch to checkout after clone")),OPT_BOOL(0,"full-clone",&full_clone,N_("when cloning, create full working directory")),+OPT_BOOL(0,"single-branch",&single_branch,+N_("only download metadata for the branch that will "+"be checked out")),OPT_END(),};constchar*constclone_usage[]={
@@ -409,7 +412,9 @@ static int cmd_clone(int argc, const char **argv)if(set_config("remote.origin.url=%s",url)||set_config("remote.origin.fetch="-"+refs/heads/*:refs/remotes/origin/*")||+"+refs/heads/%s:refs/remotes/origin/%s",+single_branch?branch:"*",+single_branch?branch:"*")||set_config("remote.origin.promisor=true")||set_config("remote.origin.partialCloneFilter=blob:none")){res=error(_("could not configure remote in '%s'"),dir);
@@ -56,6 +56,16 @@ subdirectories outside your sparse-checkout by using `git ls-tree HEAD`. Instead of checking out the branch pointed to by the cloned repository's HEAD, check out the `<name>` branch instead.+--[no-]single-branch::+ Clone only the history leading to the tip of a single branch, either+ specified by the `--branch` option or the primary branch remote's+ `HEAD` points at.+++Further fetches into the resulting repository will only update the+remote-tracking branch for the branch this option was used for the initial+cloning. If the HEAD at the remote did not point at any branch when+`--single-branch` clone was made, no remote-tracking branch is created.+ --[no-]full-clone:: A sparse-checkout is initialized by default. This behavior can be turned off via `--full-clone`.
From: Derrick Stolee via GitGitGadget <hidden> Date: 2021-09-08 19:25:10
From: Derrick Stolee <redacted>
Note: this subcommand is provided primarily for backwards-compatibility,
for existing Scalar uses. It is mostly just a shim for `git
maintenance`, mapping task names from the way Scalar called them to the
way Git calls them.
The reason why those names differ? The background maintenance was first
implemented in Scalar, and when it was contributed as a patch series
implementing the `git maintenance` command, reviewers suggested better
names, those suggestions were accepted before the patches were
integrated into core Git.
Signed-off-by: Derrick Stolee <redacted>
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 64 +++++++++++++++++++++++++++++++++++++++
contrib/scalar/scalar.txt | 19 ++++++++++++
2 files changed, 83 insertions(+)
@@ -490,6 +490,69 @@ static int cmd_register(int argc, const char **argv)returnregister_dir();}+staticintcmd_run(intargc,constchar**argv)+{+structoptionoptions[]={+OPT_END(),+};+struct{+constchar*arg,*task;+}tasks[]={+{"config",NULL},+{"commit-graph","commit-graph"},+{"fetch","prefetch"},+{"loose-objects","loose-objects"},+{"pack-files","incremental-repack"},+{NULL,NULL}+};+structstrbufbuf=STRBUF_INIT;+constchar*usagestr[]={NULL,NULL};+inti;++strbuf_addstr(&buf,N_("scalar run <task> [<enlistment>]\nTasks:\n"));+for(i=0;tasks[i].arg;i++)+strbuf_addf(&buf,"\t%s\n",tasks[i].arg);+usagestr[0]=buf.buf;++argc=parse_options(argc,argv,NULL,options,+usagestr,0);++if(!argc)+usage_with_options(usagestr,options);++if(!strcmp("all",argv[0])){+i=-1;+}else{+for(i=0;tasks[i].arg&&strcmp(tasks[i].arg,argv[0]);i++)+;/* keep looking for the task */++if(i>0&&!tasks[i].arg){+error(_("no such task: '%s'"),argv[0]);+usage_with_options(usagestr,options);+}+}++argc--;+argv++;+setup_enlistment_directory(argc,argv,usagestr,options,NULL);+strbuf_release(&buf);++if(i==0)+returnregister_dir();++if(i>0)+returnrun_git("maintenance","run",+"--task",tasks[i].task,NULL);++if(register_dir())+return-1;+for(i=1;tasks[i].arg;i++)+if(run_git("maintenance","run",+"--task",tasks[i].task,NULL))+return-1;+return0;+}+staticintremove_deleted_enlistment(structstrbuf*path){intres=0;
@@ -97,6 +98,24 @@ unregister [<enlistment>]:: Remove the specified repository from the list of repositories registered with Scalar and stop the scheduled background maintenance.+Run+~~~++scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]::+ Run the given maintenance task (or all tasks, if `all` was specified).+ Except for `all` and `config`, this subcommand simply hands off to+ linkgit:git-maintenance[1] (mapping `fetch` to `prefetch` and+ `pack-files` to `incremental-repack`).+++These tasks are run automatically as part of the scheduled maintenance,+as soon as the repository is registered with Scalar. It should therefore+not be necessary to run this subcommand manually.+++The `config` task is specific to Scalar and configures all those+opinionated default settings that make Git work more efficiently with+large repositories. As this task is run as part of `scalar clone`+automatically, explicit invocations of this task are rarely needed.+ SEE ALSO -------- linkgit:git-clone[1], linkgit:git-maintenance[1].
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-09-08 19:25:13
From: Johannes Schindelin <redacted>
This comes in handy during Scalar upgrades, or when config settings were
messed up by mistake.
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 79 +++++++++++++++++++++-----------
contrib/scalar/scalar.txt | 8 ++++
contrib/scalar/t/t9099-scalar.sh | 8 ++++
3 files changed, 67 insertions(+), 28 deletions(-)
@@ -116,6 +117,13 @@ opinionated default settings that make Git work more efficiently with large repositories. As this task is run as part of `scalar clone` automatically, explicit invocations of this task are rarely needed.+Reconfigure+~~~~~~~~~~~++After a Scalar upgrade, or when the configuration of a Scalar enlistment+was somehow corrupted or changed by mistake, this subcommand allows to+reconfigure the enlistment.+ SEE ALSO -------- linkgit:git-clone[1], linkgit:git-maintenance[1].
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-09-08 19:25:14
From: Johannes Schindelin <redacted>
After a Scalar upgrade, it can come in really handy if there is an easy
way to reconfigure all Scalar enlistments. This new option offers this
functionality.
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 61 ++++++++++++++++++++++++++++++--
contrib/scalar/scalar.txt | 9 +++--
contrib/scalar/t/t9099-scalar.sh | 3 ++
3 files changed, 67 insertions(+), 6 deletions(-)
@@ -494,22 +494,77 @@ static int cmd_register(int argc, const char **argv)returnregister_dir();}+staticintget_scalar_repos(constchar*key,constchar*value,void*data)+{+structstring_list*list=data;++if(!strcmp(key,"scalar.repo"))+string_list_append(list,value);++return0;+}+staticintcmd_reconfigure(intargc,constchar**argv){+intall=0;structoptionoptions[]={+OPT_BOOL('a',"all",&all,+N_("reconfigure all registered enlistments")),OPT_END(),};constchar*constusage[]={-N_("scalar reconfigure [<enlistment>]"),+N_("scalar reconfigure [--all | <enlistment>]"),NULL};+structstring_listscalar_repos=STRING_LIST_INIT_DUP;+inti,res=0;+structrepositoryr={NULL};+structstrbufcommondir=STRBUF_INIT,gitdir=STRBUF_INIT;argc=parse_options(argc,argv,NULL,options,usage,0);-setup_enlistment_directory(argc,argv,usage,options,NULL);+if(!all){+setup_enlistment_directory(argc,argv,usage,options,NULL);++returnset_recommended_config(1);+}++if(argc>0)+usage_msg_opt(_("--all or <enlistment>, but not both"),+usage,options);++git_config(get_scalar_repos,&scalar_repos);-returnset_recommended_config(1);+for(i=0;i<scalar_repos.nr;i++){+constchar*dir=scalar_repos.items[i].string;++strbuf_reset(&commondir);+strbuf_reset(&gitdir);++if(chdir(dir)<0){+warning_errno(_("could not switch to '%s'"),dir);+res=-1;+}elseif(discover_git_directory(&commondir,&gitdir)<0){+warning_errno(_("git repository gone in '%s'"),dir);+res=-1;+}else{+git_config_clear();++the_repository=&r;+r.commondir=commondir.buf;+r.gitdir=gitdir.buf;++if(set_recommended_config(1)<0)+res=-1;+}+}++string_list_clear(&scalar_repos,1);+strbuf_release(&commondir);+strbuf_release(&gitdir);++returnres;}staticintcmd_run(intargc,constchar**argv)
@@ -32,8 +32,8 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment will be identical to the worktree. The `scalar` command implements various subcommands, and different options-depending on the subcommand. With the exception of `clone` and `list`, all-subcommands expect to be run in an enlistment.+depending on the subcommand. With the exception of `clone`, `list` and+`reconfigure --all`, all subcommands expect to be run in an enlistment. COMMANDS --------
@@ -124,6 +124,9 @@ After a Scalar upgrade, or when the configuration of a Scalar enlistment was somehow corrupted or changed by mistake, this subcommand allows to reconfigure the enlistment.+With the `--all` option, all enlistments currently registered with Scalar+will be reconfigured. Use this option after each Scalar upgrade.+ SEE ALSO -------- linkgit:git-clone[1], linkgit:git-maintenance[1].
From: Matthew John Cheetham via GitGitGadget <hidden> Date: 2021-09-08 19:25:18
From: Matthew John Cheetham <redacted>
Delete an enlistment by first unregistering the repository and then
deleting the enlistment directory (usually the directory containing the
worktree `src/` directory).
On Windows, if the current directory is inside the enlistment's
directory, change to the parent of the enlistment directory, to allow us
to delete the enlistment (directories used by processes e.g. as current
working directories cannot be deleted on Windows).
Co-authored-by: Victoria Dye [off-list ref]
Signed-off-by: Matthew John Cheetham <redacted>
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 55 ++++++++++++++++++++++++++++++++
contrib/scalar/scalar.txt | 8 +++++
contrib/scalar/t/t9099-scalar.sh | 9 ++++++
3 files changed, 72 insertions(+)
@@ -127,6 +128,13 @@ reconfigure the enlistment. With the `--all` option, all enlistments currently registered with Scalar will be reconfigured. Use this option after each Scalar upgrade.+Delete+~~~~~~++delete <enlistment>::+ This subcommand lets you delete an existing Scalar enlistment from your+ local file system, unregistering the repository.+ SEE ALSO -------- linkgit:git-clone[1], linkgit:git-maintenance[1].
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-09-08 19:25:19
From: Johannes Schindelin <redacted>
The .NET version of Scalar has a `version` command. This was necessary
because it was versioned independently of Git.
Since Scalar is now tightly coupled with Git, it does not make sense for
them to show different versions. Therefore, it shows the same output as
`git version`. For backwards-compatibility with the .NET version,
`scalar version` prints to `stderr`, though (`git version` prints to
`stdout` instead).
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 39 +++++++++++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)
From: Johannes Schindelin via GitGitGadget <hidden> Date: 2021-09-08 19:25:20
From: Johannes Schindelin <redacted>
The `git` executable has these two very useful options:
-C <directory>:
switch to the specified directory before performing any actions
-c <key>=<value>:
temporarily configure this setting for the duration of the
specified scalar subcommand
With this commit, we teach the `scalar` executable the same trick.
Signed-off-by: Johannes Schindelin <redacted>
---
contrib/scalar/scalar.c | 22 +++++++++++++++++++++-
contrib/scalar/scalar.txt | 10 ++++++++++
2 files changed, 31 insertions(+), 1 deletion(-)
@@ -806,6 +806,25 @@ int cmd_main(int argc, const char **argv)structstrbufscalar_usage=STRBUF_INIT;inti;+while(argc>1&&*argv[1]=='-'){+if(!strcmp(argv[1],"-C")){+if(argc<3)+die(_("-C requires a <directory>"));+if(chdir(argv[2])<0)+die_errno(_("could not change to '%s'"),+argv[2]);+argc-=2;+argv+=2;+}elseif(!strcmp(argv[1],"-c")){+if(argc<3)+die(_("-c requires a <key>=<value> argument"));+git_config_push_parameter(argv[2]);+argc-=2;+argv+=2;+}else+break;+}+if(argc>1){argv++;argc--;
@@ -36,6 +36,16 @@ The `scalar` command implements various subcommands, and different options depending on the subcommand. With the exception of `clone`, `list` and `reconfigure --all`, all subcommands expect to be run in an enlistment.+The following options can be specified _before_ the subcommand:++-C <directory>::+ Before running the subcommand, change the working directory. This+ option imitates the same option of linkgit:git[1].++-c <key>=<value>::+ For the duration of running the specified subcommand, configure this+ setting. This option imitates the same option of linkgit:git[1].+ COMMANDS --------