A Python script to put CTAN into git (from DVDs)

2 messages, 2 authors, 2016-06-15 · open the first message on its own page

A Python script to put CTAN into git (from DVDs)

From: Jonathan Fine <hidden>
Date: 2016-06-15 22:52:23

Hi

This it to let you know that I'm writing (in Python) a script that 
places the content of CTAN into a git repository.
     https://bitbucket.org/jfine/python-ctantools

I'm working from the TeX Collection DVDs that are published each year by 
the TeX user groups, which contain a snapshot of CTAN (about 100,000 
files occupying 4Gb), which means I have to unzip folders and do a few 
other things.

CTAN is the Comprehensive TeX Archive Network.  CTAN keeps only the 
latest version of each file, but old CTAN snapshots will provide many 
earlier versions.

I'm working on putting old CTAN files into modern version control. 
Martin Scharrer is working in the other direction.  He's putting new 
files added to CTAN into Mercurial.
     http://ctanhg.scharrer-online.de/

My script works already as a proof of concept, but needs more work (and 
documentation) before it becomes useful.  I've requested that follow up 
goes to comp.text.tex.

Longer terms goals are git as
* http://en.wikipedia.org/wiki/Content-addressable_storage
* a resource editing and linking system

If you didn't know, a git tree is much like an immutable JSON object, 
except that it does not have arrays or numbers.

If my project interests you, reply to this message or contact me 
directly (or both).

-- 
Jonathan

Re: A Python script to put CTAN into git (from DVDs)

From: Jakub Narebski <hidden>
Date: 2016-06-15 22:52:23

Jonathan Fine [off-list ref] writes:
Hi

This it to let you know that I'm writing (in Python) a script that
places the content of CTAN into a git repository.
     https://bitbucket.org/jfine/python-ctantools
I hope that you meant "repositories" (plural) here, one per tool,
rather than putting all of CTAN into single Git repository.
 
I'm working from the TeX Collection DVDs that are published each year
by the TeX user groups, which contain a snapshot of CTAN (about
100,000 files occupying 4Gb), which means I have to unzip folders and
do a few other things.
There is 'contrib/fast-import/import-zips.py' in git.git repository.
If you are not using it, or its equivalent, it might be worth checking
out.
 
CTAN is the Comprehensive TeX Archive Network.  CTAN keeps only the
latest version of each file, but old CTAN snapshots will provide many
earlier versions.
There was similar effort done in putting CPAN (Comprehensive _Perl_
Archive Network) in Git, hosting repositories on GitHub[1], by the name
of gitPAN, see e.g.:

  "The gitPAN Import is Complete"
  http://perlisalive.com/articles/36
 
[1]: https://github.com/gitpan
I'm working on putting old CTAN files into modern version
control. Martin Scharrer is working in the other direction.  He's
putting new files added to CTAN into Mercurial.
     http://ctanhg.scharrer-online.de/
Nb. thanks to tools such as git-hg and fast-import / fast-export
we have quite good interoperability and convertability between
Git and Mercurial.

P.S. I'd point to reposurgeon tool, which can be used to do fixups
after import, but it would probably won't work on such large (set of)
repositories.

P.P.S. Can you forward it to comp.text.tex?
-- 
Jakub Narębski
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help