so I'm trying to convert KDE's translation helper scripts to support git
repositories for kde modules. the translators themselves are staying in svn
for now, but we want to start migrating kde modules over to git.
I've hit a bit of a problem, though.
inside the translators' folder, there's a folder called documentation/ - and
inside that is a bunch of svn externals pointing to the doc/ folder of every
kde module. (this is, then, all of kde's english documentation.)
translators occasionally need to run a script that reads from the
documentation/ folder.
now, what happens when one of those modules switches to git? first, the svn
external goes away. that's ok, I can work around that by having the update_xml
script (the one translators need to run) fetch the doc folder itself, right?
but where does it fetch it from? doc/ is now a subdirectory in a git
repository, and git doesn't support checking out only one folder, not even if
all I want to do is read the files in it.
checking out the entire git repo isn't really feasible, because once KDE
finishes switching to git that'll mean checking out all of KDE.
I don't think that turning every doc folder into a submodule is feasible,
either - they sound kinda awkward to work with, and I can imagine lots of
people getting confused and messing them up...
right now all I've thought of is one ugly hack: have a server that checks out
all the kde git repos, pulls daily, copies all the doc/ folders into a
documentation folder, and offers that folder up on the interwebs so that
update_xml can rsync from it or download a tgz of it or something.
there appear to be lots of images in the documentation, so it's not a small
download - 200mb and growing. it still hasn't finished downloading all the
externals...
I'm kinda wondering if there'd be a way to use git-filter-branch to make a repo
that only tracks the doc/ folder for a module - but I've no idea whether it'd
have to be recreated from scratch every time someone changes something in the
real repo's doc/
can anyone think of a less ugly solution?
what are the chances of git supporting this kind of partial checkout someday?
--
This message brought to you by eevil bananas and the number 3.
www.chani3.com
From: Johan Herland <hidden> Date: 2016-06-15 22:46:49
On Saturday 23 May 2009, Chani wrote:
[...]
right now all I've thought of is one ugly hack: have a server that checks
out all the kde git repos, pulls daily, copies all the doc/ folders into
a documentation folder, and offers that folder up on the interwebs so
that update_xml can rsync from it or download a tgz of it or something.
there appear to be lots of images in the documentation, so it's not a
small download - 200mb and growing. it still hasn't finished downloading
all the externals...
Do you need the doc/ folders from _all_ kde git repos, or just from those
repos that you have currently checked out? In the latter case, you could
solve this by adding symlinks to all the doc/ folders inside the
documentation/ folder, and then make sure the software that traverse the
documentation/ folder recognize and skips symlinks. Of course, this won't
work if the translations project need _all_ doc/ folders accessible, but not
all the kde git repos.
I'm kinda wondering if there'd be a way to use git-filter-branch to make
a repo that only tracks the doc/ folder for a module - but I've no idea
whether it'd have to be recreated from scratch every time someone changes
something in the real repo's doc/
can anyone think of a less ugly solution?
what are the chances of git supporting this kind of partial checkout
someday?
Check out git-subtree. It can split out a subdirectory into its own repo,
and re-integrate it back into the "parent" repo at a later date. git-subtree
has been posted as a patch to this list a couple of times without much
response, but it looks like an interesting alternative to submodules:
http://alumnit.ca/~apenwarr/log/?m=200904#30
If a lot of people find git-subtree useful, who knows, it might be included
in a future git version.
Have fun! :)
...Johan
--
Johan Herland, [off-list ref]
www.herland.net
[...]
right now all I've thought of is one ugly hack: have a server that checks
out all the kde git repos, pulls daily, copies all the doc/ folders into
a documentation folder, and offers that folder up on the interwebs so
that update_xml can rsync from it or download a tgz of it or something.
there appear to be lots of images in the documentation, so it's not a
small download - 200mb and growing. it still hasn't finished downloading
all the externals...
Do you need the doc/ folders from _all_ kde git repos, or just from those
repos that you have currently checked out? In the latter case, you could
solve this by adding symlinks to all the doc/ folders inside the
documentation/ folder, and then make sure the software that traverse the
documentation/ folder recognize and skips symlinks. Of course, this won't
work if the translations project need _all_ doc/ folders accessible, but
not all the kde git repos.
nope, the translators may not have checked out *any* of them but the script
they want to run needs *all* the docs. :( however, I've been told they also
don't want to have to change their workflow in any way at all no matter how
small, so we may be stuck in svn-land anyways, because you can't make an svn
external out of something that's not in svn, and having anything other than
svn externals would change their workflow :(
quoted
I'm kinda wondering if there'd be a way to use git-filter-branch to make
a repo that only tracks the doc/ folder for a module - but I've no idea
whether it'd have to be recreated from scratch every time someone changes
something in the real repo's doc/
can anyone think of a less ugly solution?
what are the chances of git supporting this kind of partial checkout
someday?
Check out git-subtree. It can split out a subdirectory into its own repo,
and re-integrate it back into the "parent" repo at a later date.
git-subtree has been posted as a patch to this list a couple of times
without much response, but it looks like an interesting alternative to
submodules: http://alumnit.ca/~apenwarr/log/?m=200904#30
If a lot of people find git-subtree useful, who knows, it might be included
in a future git version.
looks interesting. might have been a solution until I heard about this
requirement to not change workflow at all. :/
however, my friend told me about a project to make a git-svnserver that serves
git repos as svn repos, and *that* would allow the translators to stay where
they are without holding everyone else there too. know anything about that?
mm, google turns up an email from someone claiming they have a partial
implementation in python...
--
This message brought to you by eevil bananas and the number 3.
www.chani3.com
On Sun, May 24, 2009 at 7:00 AM, Chani [off-list ref] wrote:
checking out the entire git repo isn't really feasible, because once KDE
finishes switching to git that'll mean checking out all of KDE.
I don't think that turning every doc folder into a submodule is feasible,
either - they sound kinda awkward to work with, and I can imagine lots of
people getting confused and messing them up...
If you don't mind cloning the entire git repo (which may be quite
large in KDE case, I guess), then you may want to try "sparse
checkout". The patch series is available as a topic branch "tp/sco" in
this branch:
http://repo.or.cz/w/git/pclouds.git?a=shortlog;h=refs/heads/inst
Or I can send the patch series to you. It lets you specify what part
of the tree you want to checkout.
--
Duy
From: Thomas Adam <hidden> Date: 2016-06-15 22:46:49
2009/5/24 Nguyen Thai Ngoc Duy [off-list ref]:
On Sun, May 24, 2009 at 7:00 AM, Chani [off-list ref] wrote:
quoted
checking out the entire git repo isn't really feasible, because once KDE
finishes switching to git that'll mean checking out all of KDE.
I don't think that turning every doc folder into a submodule is feasible,
either - they sound kinda awkward to work with, and I can imagine lots of
people getting confused and messing them up...
If you don't mind cloning the entire git repo (which may be quite
large in KDE case, I guess), then you may want to try "sparse
checkout". The patch series is available as a topic branch "tp/sco" in
this branch:
http://repo.or.cz/w/git/pclouds.git?a=shortlog;h=refs/heads/inst
Or I can send the patch series to you. It lets you specify what part
of the tree you want to checkout.
I recall following this series when you first announced it. Do you
have any plans to try and get it included in mainline? ISTR there was
some more work needed to be done on it before then.
-- Thomas Adam
On Sat, May 23, 2009 at 8:26 PM, Chani [off-list ref] wrote:
On May 23, 2009 16:34:53 Johan Herland wrote:
quoted
On Saturday 23 May 2009, Chani wrote:
quoted
I'm kinda wondering if there'd be a way to use git-filter-branch to make
a repo that only tracks the doc/ folder for a module - but I've no idea
whether it'd have to be recreated from scratch every time someone changes
something in the real repo's doc/
can anyone think of a less ugly solution?
what are the chances of git supporting this kind of partial checkout
someday?
Check out git-subtree. It can split out a subdirectory into its own repo,
and re-integrate it back into the "parent" repo at a later date.
git-subtree has been posted as a patch to this list a couple of times
without much response, but it looks like an interesting alternative to
submodules: http://alumnit.ca/~apenwarr/log/?m=200904#30
looks interesting. might have been a solution until I heard about this
requirement to not change workflow at all. :/
Here's something that might work. Write a script that runs on a
server somewhere. It checks out *all* of the git-managed KDE
components, including documentation (it's okay to do this from *one*
place, right?). Then it loops through all the documentation
subdirectories and splits them out into individual projects/branches
with 'git subtree split'.
You could then make a *new* project, kde-docs for example, that pulls
those new doc projects back together into a single project, using 'git
subtree merge'.
People could then checkout the single kde-docs project and get *all*
the docs, then make changes and push them back upstream to you. As
far as the average person is concerned, this is probably even easier
than svn:externals - they don't have to remember to commit in each
subdirectory separately, as git-subtree rips it all apart later. Also
note that *end users* never have to operate the git subtree command;
all that happens behind the scenes on your server.
Later, you use 'git subtree split' to rip it apart back into the
individual projects, and 'git subtree merge' those back into their
original git repositories.
Since git-subtree is my own little toy and it's designed for exactly
this kind of workflow, I'd be happy to help you debug a script for
this if you like. Probably I'll learn something.
Disclaimer: git-subtree hasn't been used by too many people yet and we
may end up finding some bugs.
Have fun,
Avery
On Mon, May 25, 2009 at 1:57 AM, Thomas Adam [off-list ref] wrote:
2009/5/24 Nguyen Thai Ngoc Duy [off-list ref]:
quoted
On Sun, May 24, 2009 at 7:00 AM, Chani [off-list ref] wrote:
quoted
checking out the entire git repo isn't really feasible, because once KDE
finishes switching to git that'll mean checking out all of KDE.
I don't think that turning every doc folder into a submodule is feasible,
either - they sound kinda awkward to work with, and I can imagine lots of
people getting confused and messing them up...
If you don't mind cloning the entire git repo (which may be quite
large in KDE case, I guess), then you may want to try "sparse
checkout". The patch series is available as a topic branch "tp/sco" in
this branch:
http://repo.or.cz/w/git/pclouds.git?a=shortlog;h=refs/heads/inst
Or I can send the patch series to you. It lets you specify what part
of the tree you want to checkout.
I recall following this series when you first announced it. Do you
have any plans to try and get it included in mainline? ISTR there was
some more work needed to be done on it before then.
It was pulled off because of big, intrusive and not well-designed patches.
http://mid.gmane.org/20081128192033.GF23984@spearce.org
I may submit it again when I get code insertions down to ~500 lines or
something like that ;-) Anyway you could always help by giving more
use cases to test against the current design.
--
Duy
Check out git-subtree. It can split out a subdirectory into its own
repo, and re-integrate it back into the "parent" repo at a later date.
git-subtree has been posted as a patch to this list a couple of times
without much response, but it looks like an interesting alternative to
submodules: http://alumnit.ca/~apenwarr/log/?m=200904#30
looks interesting. might have been a solution until I heard about this
requirement to not change workflow at all. :/
Here's something that might work. Write a script that runs on a
server somewhere. It checks out *all* of the git-managed KDE
components, including documentation (it's okay to do this from *one*
place, right?). Then it loops through all the documentation
subdirectories and splits them out into individual projects/branches
with 'git subtree split'.
I like this idea. I like the idea of git subtree in general. :)
however, we've decided to go the "copy all documentation into svn" route for
now (so that the translators can keep using only svn).
I'll be back when we run into the next svn-external issue (there's sure to be
more). :)
--
This message brought to you by eevil bananas and the number 3.
www.chani3.com
From: Aidan Van Dyk <hidden> Date: 2016-06-15 22:46:49
If all you're looking for is a "single set of read-only docs" conglomeration of
all the KDE submodules projects documentation so you can translate them, what
about using "archive"? Since archive takes a "tree-ish", and work on remote
upload-pack servers (if it's enabled), you can get an un-versioned "remote
partial checkout" with something like do things like:
git archive --format=tar --prefix=git/Documentation/ --remote=git://repo.or.cz/git.git HEAD:Documentation | tar tf - | head
git archive --format=tar --prefix=/opt/git/share/man/ --remote=git://repo.or.cz/git.git man | tar tf -
So, this could be simply scripted to give you a repo with *all* modules docs/ inside a single repo...
for m in $MODULES
do
git archive --format=tar --prefix="$m/$DOCDIR/" --remote=$REMOTE_BASE/$m $BRANCH:$DOCDIR | tar xf -
git add $m/$DOCDIR
git commit $m/$DOCDIR -m "$m: Automatic import"
done
Do that in a central location an dlet translators pull that... Everythings
"duplicated", but git's object store will help out there... You loose the
history of the original docs, but for translations, you you're not translating
the history anyways, your'e translation the docs at a point in time...
Or just dump all thd docs in a .gitignore'd subdirctory and have translators
run a script to update them.
a.
On Sat, May 23, 2009 at 8:26 PM, Chani [off-list ref] wrote:
quoted
On May 23, 2009 16:34:53 Johan Herland wrote:
quoted
On Saturday 23 May 2009, Chani wrote:
quoted
I'm kinda wondering if there'd be a way to use git-filter-branch to make
a repo that only tracks the doc/ folder for a module - but I've no idea
whether it'd have to be recreated from scratch every time someone changes
something in the real repo's doc/
can anyone think of a less ugly solution?
what are the chances of git supporting this kind of partial checkout
someday?
Here's something that might work. Write a script that runs on a
server somewhere. It checks out all of the git-managed KDE
components, including documentation (it's okay to do this from one
place, right?). Then it loops through all the documentation
subdirectories and splits them out into individual projects/branches
with 'git subtree split'.
You could then make a new project, kde-docs for example, that pulls
those new doc projects back together into a single project, using 'git
subtree merge'.
People could then checkout the single kde-docs project and get all
the docs, then make changes and push them back upstream to you. As
far as the average person is concerned, this is probably even easier
than svn:externals - they don't have to remember to commit in each
subdirectory separately, as git-subtree rips it all apart later. Also
note that *end users* never have to operate the git subtree command;
all that happens behind the scenes on your server.
Later, you use 'git subtree split' to rip it apart back into the
individual projects, and 'git subtree merge' those back into their
original git repositories.
Since git-subtree is my own little toy and it's designed for exactly
this kind of workflow, I'd be happy to help you debug a script for
this if you like. Probably I'll learn something.
Disclaimer: git-subtree hasn't been used by too many people yet and we
may end up finding some bugs.
Have fun,
Avery