Derrick Stolee [off-list ref] writes:
Optimization Factors
--------------------
There are multiple factors involved that we wish to optimize. Here is my
understanding of the most-important factors and how they interact:
### Value
The main reason to include Scalar within the Git codebase is to make it
available to more users. This should have downstream effects based on user
feedback about what they like or dislike about the Scalar CLI that can
inform future features in core Git that simplify repository management at
scale.
In other words, Scalar wants to piggyback on Git's popularity, yet
wants to retain total control of how its end-user experience should
look like, eating the cake and having it at the same time?
I do not think it is fair to proceed with this discussion without
having "Scalar lives in its own tree, uses this mailing list to
discuss how it should evolve, drawing expertise from the
contributors found here, but otherwise would stay to be a spearate
project" as another option and debating its pros-and-cons, next to
the spectrum among contrib/scalar, scalar/, and scalar.c approaches.
IIUC, cgit has one-directional coupling to depend on libgit.a just
like the Scalar wants to link with it---I am not saying Scalar should
do the same, but just raising it as an example that you do *not* have
to live in my tree anywhere to be able to link with libgit.a; that
would probably count as an "**Option 0:** A separate project that
borrows from Git by having it as a submodule".
I can think of only one obvious advantage going that route, and that
is why I said "I am not saying Scalar should do the same", which is
that the Scalar project will then retain total control over how it
is built and how its end-user experience should stay the same.
I do not think it is an option to refuse to take any input from
others on this list on parts of what you have already done, if you
want to be in my tree, anywhere, not even in contrib/.
Just like we say "the plumbing command X has behaved this way
forever, and we cannot break the backward compatibility" to some
possible changes to any of our tools, however, "Scalar has been used
with that subcommand behaving this particular way even before the
code was donated to Git project, so we cannot lightly change it"
would equally be a good input when we evaluate when some things in
it may need to be changed, so I do not think it would pose a great
backward compatibility issue (after all, we do not change things
lightly just for the sake of changing it).
But approaching our developers with a new piece of code, saying "I
want to piggyback on your popularity. Among the stuff given to you,
some are out of limit and you may not even discuss about modifying
it" upfront, is a problematic attitude.
Now, the goal from purely Git project's point of view would be to
improve Git's user experience, not necessarily Scalar's, by exposing
more of our users to the way its opinionated (which is *not*
necessarily a bad thing) decisions work to help its users, and our
developers to the scalar codebase. Some of our developers may not
even agree with the design decisions Scalar made and that is
expected and is fine; based on such a reaction, the lesson Git
learns from Scalar to solve a similar issue may turn out to be solve
it differently.
I do not know how much Options 1-3 would help such cross pollination
more over the Option 0, offhand, though.
Possible approaches to land in our final target
-----------------------------------------------
Here I sense the "final target" is somehow Scalar is delivered to
end-users through my tree and being maintained there, but from my
point of view, that is merely an early step. Wasn't the final
target for "git" to learn from Scalar's successes and mistakes and
give native support to itches Scalar wanted to scratch?
And if that is indeed the goal, when it is achieved, Scalar would
become a historical curiosity just like Cogito is today. With
Cogito, we cross-pollinated just fine without having both in the
same tree at all, and when Cogito outlived its usefulness, we did
not have to remove anything from our tree. With cgit, we did not
cross-pollinate at all, unfortunately, and gitweb more or less stays
the same way as it was years ago, and cgit is still a project with a
useful product. We can reach both extremes even if we take Option
0. But it would have been messy for something like Cogito, which
was an early UI and workflow experiment, if we took a more
integrated approach.
**Phase 1.** Keep code in contrib/ while we contribute the Scalar features.
Since the current patches on the mailing list are not feature complete, it
can be beneficial to move forward with the code in `contrib/scalar/` until
we reach feature parity. At that point, we could move the source into its
final resting place.
This leans into the fact that `contrib/` contains "interesting tools...
maybe even experimental ones". While Scalar is not feature complete, it
can be isolated as experimental here until it is ready for promotion to
non-experimental.
I personally have no problem with this approach, but with less
scrutiny, being in contrib/ longer, with "some stuff are not subject
to discussion" attitude, would inevitably lower the code quality of
the product, and it is dubious it would become "ready for promotion"
ever. The contrib/ hierarchy, especially in early days when we
wanted a way to grow the ecosystem, as a place to host "these are
done externally, and we only carry them for convenience", was a
successful model, but I do not think of anything that successfully
used contrib/ as a nursery to later graduate to the core and to be
seen as the same quality as other things that started their life in
core. I am pessimistic to the model that uses contrib/ as such a
place.
Depending on the final goal, we could drop some of the work that is currently
built within the `contrib/scalar/` directory, such as `-c`/`-C` parsing or
documentation builds. These features would be reimplemented in the new
location, so we can prevent that duplicate effort if we have a different
final location in mind.
**Phase 2.** Establish community standards on optional components
While the work in `contrib/scalar/` continues to reach feature parity, we
can work as a community to set some ground rules about these kinds of
optional features (depending on how "optional" we land). This can include
standards such as how the files are laid out in the repository and how
they interact with the `Makefile`. Whatever we do for Scalar is likely to
form an example for a future contribution that we can't predict today.
Many of the questions we are asking today can be written for posterity:
* What is an appropriate level of coupling with the Git codebase?
* Where should code custom to the component live?
* How should documentation and tests be organized?
* How do we initialize builds of a component?
* Who is responsible for supporting the component? Who fixes the bugs?
* Who is responsible for reviewing changes to the component?
I am afraid that the answer to the first four are "it depends" (but
the remaining two would fall out naturally out of them).
Thanks. My brain is fried with the -rc work, rereading the material
that went to -rc0 since the last release, so I may have more to add
later, but for now, the above is what came to my mind.
Hi Junio,
On Mon, 1 Nov 2021, Junio C Hamano wrote:
Derrick Stolee [off-list ref] writes:
quoted
Optimization Factors
--------------------
There are multiple factors involved that we wish to optimize. Here is
my understanding of the most-important factors and how they interact:
### Value
The main reason to include Scalar within the Git codebase is to make
it available to more users. This should have downstream effects based
on user feedback about what they like or dislike about the Scalar CLI
that can inform future features in core Git that simplify repository
management at scale.
In other words, Scalar wants to piggyback on Git's popularity, yet wants
to retain total control of how its end-user experience should look like,
eating the cake and having it at the same time?
I do not think it is fair to proceed with this discussion without having
"Scalar lives in its own tree, uses this mailing list to discuss how it
should evolve, drawing expertise from the contributors found here, but
otherwise would stay to be a spearate project" as another option and
debating its pros-and-cons, next to the spectrum among contrib/scalar,
scalar/, and scalar.c approaches.
If that was indeed the intention with Scalar, I would totally agree.
We don't want Scalar to piggy-back onto Git. In fact, we don't even want
it to be separate from Git, which is why we are contributing it here. I
firmly believe that Scalar has a lot of value and will benefit the Git
project and community. That is my motivation behind contributing these
patches.
IIUC, cgit has one-directional coupling to depend on libgit.a just like
the Scalar wants to link with it---I am not saying Scalar should do the
same, but just raising it as an example that you do *not* have to live
in my tree anywhere to be able to link with libgit.a; that would
probably count as an "**Option 0:** A separate project that borrows from
Git by having it as a submodule".
I did consider keeping Scalar/C in its own repository, and I, too, had
cgit in mind (and cinnabar). In fact, Scalar/C does live in its own
repository right now, and would remain there if the community doesn’t want
it: in the microsoft/git fork of git/git.
However, just like partial clone, scheduled maintenance, sparse-checkout
cone mode and sparse-index, the remainder of Scalar can benefit Git by
moving its functionality to `libgit.a`, or reimplementing it there.
I can think of only one obvious advantage going that route, and that is
why I said "I am not saying Scalar should do the same", which is that
the Scalar project will then retain total control over how it is built
and how its end-user experience should stay the same.
I do not think it is an option to refuse to take any input from others
on this list on parts of what you have already done, if you want to be
in my tree, anywhere, not even in contrib/.
Just like we say "the plumbing command X has behaved this way forever,
and we cannot break the backward compatibility" to some possible changes
to any of our tools, however, "Scalar has been used with that subcommand
behaving this particular way even before the code was donated to Git
project, so we cannot lightly change it" would equally be a good input
when we evaluate when some things in it may need to be changed, so I do
not think it would pose a great backward compatibility issue (after all,
we do not change things lightly just for the sake of changing it).
That's a really good point.
But approaching our developers with a new piece of code, saying "I want
to piggyback on your popularity. Among the stuff given to you, some are
out of limit and you may not even discuss about modifying it" upfront,
is a problematic attitude.
Yes, I agree, that would be a problematic attitude. I apologize for giving
you the impression that that was my attitude. I want to contribute Scalar
to Git because I believe that Git will benefit from it, not because I want
to abuse Git’s popularity to advance Scalar.
Now, the goal from purely Git project's point of view would be to
improve Git's user experience, not necessarily Scalar's, by exposing
more of our users to the way its opinionated (which is *not* necessarily
a bad thing) decisions work to help its users, and our developers to the
scalar codebase. Some of our developers may not even agree with the
design decisions Scalar made and that is expected and is fine; based on
such a reaction, the lesson Git learns from Scalar to solve a similar
issue may turn out to be solve it differently.
I do not know how much Options 1-3 would help such cross pollination
more over the Option 0, offhand, though.
I am still of the opinion that having the code in the same tree will make
cross pollination much easier.
quoted
Possible approaches to land in our final target
-----------------------------------------------
Here I sense the "final target" is somehow Scalar is delivered to
end-users through my tree and being maintained there, but from my point
of view, that is merely an early step. Wasn't the final target for
"git" to learn from Scalar's successes and mistakes and give native
support to itches Scalar wanted to scratch?
I think we are very much in alignment here, and I believe Stolee's goal
was to reach such an alignment.
And if that is indeed the goal, when it is achieved, Scalar would become
a historical curiosity just like Cogito is today. With Cogito, we
cross-pollinated just fine without having both in the same tree at all,
and when Cogito outlived its usefulness, we did not have to remove
anything from our tree. With cgit, we did not cross-pollinate at all,
unfortunately, and gitweb more or less stays the same way as it was
years ago, and cgit is still a project with a useful product. We can
reach both extremes even if we take Option 0. But it would have been
messy for something like Cogito, which was an early UI and workflow
experiment, if we took a more integrated approach.
Scalar is more similar to cgit than to gitweb or Cogito in this context:
gitweb and Cogito are scripted users of Git, whereas Scalar and cgit link
to libgit.a, are written in C, and it is easy to refactor code from Scalar
into libgit.a. That was never the case with gitweb nor with Cogito.
quoted
**Phase 1.** Keep code in contrib/ while we contribute the Scalar
features.
Since the current patches on the mailing list are not feature
complete, it can be beneficial to move forward with the code in
`contrib/scalar/` until we reach feature parity. At that point, we
could move the source into its final resting place.
This leans into the fact that `contrib/` contains "interesting
tools... maybe even experimental ones". While Scalar is not feature
complete, it can be isolated as experimental here until it is ready
for promotion to non-experimental.
I personally have no problem with this approach, but with less scrutiny,
being in contrib/ longer, with "some stuff are not subject to
discussion" attitude, would inevitably lower the code quality of the
product, and it is dubious it would become "ready for promotion" ever.
To be clear, I'm not pushing for promotion of Scalar to a top-level
command delivered via the Git project at all. I _might_ push for it in the
future, _iff_ a wider audience has used Scalar _and_ finds it useful.
In any case, pushing for Scalar to be such a top-level command would be
premature at this stage: to avoid overwhelming reviewers, the Scalar patch
series is not yet feature-complete, there are other patch series I intend
to contribute on top, when the first patch series has settled down.
The contrib/ hierarchy, especially in early days when we wanted a way to
grow the ecosystem, as a place to host "these are done externally, and
we only carry them for convenience", was a successful model, but I do
not think of anything that successfully used contrib/ as a nursery to
later graduate to the core and to be seen as the same quality as other
things that started their life in core. I am pessimistic to the model
that uses contrib/ as such a place.
After seeing `vcs-svn`'s fate, I understand the trepidation. I, too, would
have loved to see it being cooked to perfection and then graduate.
Even so, I targeted `contrib/` with my patch series because
`contrib/README` says:
The intention is to keep interesting tools around git here, maybe
even experimental ones, to give users an easier access to them,
and to give tools wider exposure, so that they can be improved
faster.
Does this README's intent still apply, or has the purpose of contrib/
changed?
quoted
Depending on the final goal, we could drop some of the work that is
currently built within the `contrib/scalar/` directory, such as
`-c`/`-C` parsing or documentation builds. These features would be
reimplemented in the new location, so we can prevent that duplicate
effort if we have a different final location in mind.
quoted
**Phase 2.** Establish community standards on optional components
While the work in `contrib/scalar/` continues to reach feature parity,
we can work as a community to set some ground rules about these kinds
of optional features (depending on how "optional" we land). This can
include standards such as how the files are laid out in the repository
and how they interact with the `Makefile`. Whatever we do for Scalar
is likely to form an example for a future contribution that we can't
predict today. Many of the questions we are asking today can be
written for posterity:
* What is an appropriate level of coupling with the Git codebase?
* Where should code custom to the component live?
* How should documentation and tests be organized?
* How do we initialize builds of a component?
* Who is responsible for supporting the component? Who fixes the bugs?
* Who is responsible for reviewing changes to the component?
I am afraid that the answer to the first four are "it depends" (but the
remaining two would fall out naturally out of them).
Would you like me to organize the code and patches such that they more
clearly fall under your maintenance once the patches are merged? If that
is your preference, I will gladly put in the work.
When I submitted the Scalar patches, I did not want to burden you with
maintaining it, I expected to maintain Scalar going forward, but maybe
that is not your preference? I assumed it to be my maintenance burden, we
aim to support backward compatibility for existing Scalar users, after
all. Absolutely everything else, however, is up for discussion, subject to
the full scrutiny of the community and to the same standards of a
submission to the core of the project.
And I see your point about backward compatibility: The Git project has
plenty of experience with maintaining backward compatibility and _still_
improving on initial designs. I therefore can easily give up my hard “this
command-line interface must not be changed” stance.
Thanks. My brain is fried with the -rc work, rereading the material
that went to -rc0 since the last release, so I may have more to add
later, but for now, the above is what came to my mind.
Thank you for taking the time to respond, especially during the -rc cycle.
I look forward to continuing this thread when you have time.
Ciao,
Dscho
Johannes Schindelin [off-list ref] writes:
Even so, I targeted `contrib/` with my patch series because
`contrib/README` says:
The intention is to keep interesting tools around git here, maybe
even experimental ones, to give users an easier access to them,
and to give tools wider exposure, so that they can be improved
faster.
Does this README's intent still apply, or has the purpose of contrib/
changed?
The intent may still be the same, but my suspicion is that the world
has changed sufficiently to make contrib/ that is offered with such
an intent no longer is useful. In other words, "contrib/" as a
nursery may have been our dream, but the dream did not materialize
and may have outlived its usefulness.
Would you like me to organize the code and patches such that they more
clearly fall under your maintenance once the patches are merged? If that
is your preference, I will gladly put in the work.
When I submitted the Scalar patches, I did not want to burden you with
maintaining it, I expected to maintain Scalar going forward, but maybe
that is not your preference? I assumed it to be my maintenance burden, we
aim to support backward compatibility for existing Scalar users, after
all. Absolutely everything else, however, is up for discussion, subject to
the full scrutiny of the community and to the same standards of a
submission to the core of the project.
And I see your point about backward compatibility: The Git project has
plenty of experience with maintaining backward compatibility and _still_
improving on initial designs. I therefore can easily give up my hard “this
command-line interface must not be changed” stance.
It depends on how you define "maintenance".
Many parts of the system (e.g. difftool, commit-graph, pack bitmaps,
multi-pack index, sparse-index, po, and gitweb) may "fall under my
maintenance", but you can see that I am not heavily involved in any
of these areas, other than serving mostly as a traffic cop. If a
patch that touches some area comes, I may trust input from folks who
are more deeply involved in the area (e.g. input from Taylor better
than from other folks, including mine, for a patch that touches
midx) when deciding if the patch needs revising or if it is ready.
I would expect that is more or less how things would work for
anything that is in my tree. And I do not think it would be any
different for Scalar. I would expect that any bugfixes or new
features would be redirected to you folks. Ehh, not that you would
need redirecting---I'd expect you'd jump on them even _before_ I
would react ;-)