blobs (once more)

Hello,

Binary large objects. I know it has been discussed once and again but
I'd like to know if there is something new.

Some corporation hired the company I work for one year ago to develop
a large application. They imposed ClearCase as the VCS. I don't know
if you have used it but it is a pain in the ass. We have lost weeks of
development to site-replication problems, funny merges, etc. We are
trying to migrate our project to git, which we have experience with.

One very important point in this project (which is Windows only) is
putting binaries in the repository. So far, we have suceeded in not
doing that in other projects but we will need to do that in this
project.

In the Windows world, it is not unusual to use third-party libraries
which are only available in binary form. Getting them as source is not
an option because the companies developing them are not selling the
source. Moving from those binary-only dependencies to something else
is not an option either because what we are using has some unique
features, be it technical features or support features. In our
project, we have about a dozen such binaries, ranging from a few
hundred kilobytes, to a couple hundred megabytes (proprietary database
and virtualization engine).

The usual answer to the "I need to put binaries in the repository"
question has been "no, you do not". Well, we do. We are in heavy
development now, therefore today's version may depend on a certain
version of a third-party shared library (DLL) which we only can get in
binary form, and tomorrow's version may depend on the next version of
that library, and you cannot mix today's source with yesterday's
third-party DLL. I. e. to be able to use the code from 7 days ago at
11.07 AM you need "git checkout" to "return" our source AND the
binaries we were using back then. This is something ClearCase manages
satisfactorily.

I have read about:
- submodules + using different repositories once one "blob repository"
grows too much. This will be probably rejected because it is quite
contrived.
- git-annex (does not get the files in when cloning, pulling, checking
out; you need to do it manually)
- git-media (same as git-annex)
- boar (no, we do not want to use a VCS for binaries in addition to git)
- and a few more

So far the only good solution seems to be git-bigfiles but it's still
in development.

Is there any good solution for my use case, where version = sources
version + binaries version?

Thank you.

If we suceed with git here, the whole corportation (150,000+
employees, Fortune 500) may start to move to git in a year. Many
people are fed up with CC there.

-- 
Pau Garcia i Quiles
http://www.elpauer.org
(Due to my workload, I may need 10 days to answer)

Hi,

On Wed, 6 Apr 2011, Pau Garcia i Quiles wrote:

Binary large objects. I know it has been discussed once and again but 
I'd like to know if there is something new.

Some corporation hired the company I work for one year ago to develop a 
large application. They imposed ClearCase as the VCS. I don't know if 
you have used it but it is a pain in the ass. We have lost weeks of 
development to site-replication problems, funny merges, etc. We are 
trying to migrate our project to git, which we have experience with.

One very important point in this project (which is Windows only) is 
putting binaries in the repository. So far, we have suceeded in not 
doing that in other projects but we will need to do that in this 
project.

In the Windows world, it is not unusual to use third-party libraries 
which are only available in binary form. Getting them as source is not 
an option because the companies developing them are not selling the 
source. Moving from those binary-only dependencies to something else is 
not an option either because what we are using has some unique features, 
be it technical features or support features. In our project, we have 
about a dozen such binaries, ranging from a few hundred kilobytes, to a 
couple hundred megabytes (proprietary database and virtualization 
engine).

The usual answer to the "I need to put binaries in the repository" 
question has been "no, you do not". Well, we do. We are in heavy 
development now, therefore today's version may depend on a certain 
version of a third-party shared library (DLL) which we only can get in 
binary form, and tomorrow's version may depend on the next version of 
that library, and you cannot mix today's source with yesterday's 
third-party DLL. I. e. to be able to use the code from 7 days ago at 
11.07 AM you need "git checkout" to "return" our source AND the binaries 
we were using back then. This is something ClearCase manages 
satisfactorily.

I understand. The problem in your case might not be too bad, after all. 
The problem only arises when you have big files that are compressed. If 
you check in multiple versions of an uncompressed .dll file, Git will 
usually do a very good job at compressing them.

If they are compressed, what you probably need is something like a sparse 
clone, which is sort of available in the form of shallow clones, but it is 
too limited still.

Having said that, in another company I work for, they hav 20G repositories 
and they will grow larger. That is something they incurred due to 
historical reasons, and they are willing to pay the price in terms of disk 
space. Due to Git's distributed nature, they had no problems with cloning; 
they just use a local reference upon initial clone.

I have read about:
- submodules + using different repositories once one "blob repository"  
  grows too much. This will be probably rejected because it is quite 
  contrived.

I would also recommend against this, because submodules are a very weak 
part of Git.

- git-annex (does not get the files in when cloning, pulling, checking 
  out; you need to do it manually)
- git-media (same as git-annex)

Yes, this is an option, but a bit klunky.

- boar (no, we do not want to use a VCS for binaries in addition to git)

I did not know about that.

- and a few more

So far the only good solution seems to be git-bigfiles but it's still
in development.

It has stalled, apparently, but I wanted to have a look at it anyway. Will 
let you know of my findings!

Is there any good solution for my use case, where version = sources 
version + binaries version?

Thank you.

If we suceed with git here, the whole corportation (150,000+
employees, Fortune 500) may start to move to git in a year. Many
people are fed up with CC there.

Ciao,
Johannes

Johannes Schindelin venit, vidit, dixit 06.04.2011 11:25:

Hi,

On Wed, 6 Apr 2011, Pau Garcia i Quiles wrote:

quoted

Binary large objects. I know it has been discussed once and again but 
I'd like to know if there is something new.

Some corporation hired the company I work for one year ago to develop a 
large application. They imposed ClearCase as the VCS. I don't know if 
you have used it but it is a pain in the ass. We have lost weeks of 
development to site-replication problems, funny merges, etc. We are 
trying to migrate our project to git, which we have experience with.

One very important point in this project (which is Windows only) is 
putting binaries in the repository. So far, we have suceeded in not 
doing that in other projects but we will need to do that in this 
project.

In the Windows world, it is not unusual to use third-party libraries 
which are only available in binary form. Getting them as source is not 
an option because the companies developing them are not selling the 
source. Moving from those binary-only dependencies to something else is 
not an option either because what we are using has some unique features, 
be it technical features or support features. In our project, we have 
about a dozen such binaries, ranging from a few hundred kilobytes, to a 
couple hundred megabytes (proprietary database and virtualization 
engine).

The usual answer to the "I need to put binaries in the repository" 
question has been "no, you do not". Well, we do. We are in heavy 
development now, therefore today's version may depend on a certain 
version of a third-party shared library (DLL) which we only can get in 
binary form, and tomorrow's version may depend on the next version of 
that library, and you cannot mix today's source with yesterday's 
third-party DLL. I. e. to be able to use the code from 7 days ago at 
11.07 AM you need "git checkout" to "return" our source AND the binaries 
we were using back then. This is something ClearCase manages 
satisfactorily.

I understand. The problem in your case might not be too bad, after all. 
The problem only arises when you have big files that are compressed. If 
you check in multiple versions of an uncompressed .dll file, Git will 
usually do a very good job at compressing them.

If they are compressed, what you probably need is something like a sparse 
clone, which is sort of available in the form of shallow clones, but it is 
too limited still.

Having said that, in another company I work for, they hav 20G repositories 
and they will grow larger. That is something they incurred due to 
historical reasons, and they are willing to pay the price in terms of disk 
space. Due to Git's distributed nature, they had no problems with cloning; 
they just use a local reference upon initial clone.

quoted

I have read about:
- submodules + using different repositories once one "blob repository"  
  grows too much. This will be probably rejected because it is quite 
  contrived.

I would also recommend against this, because submodules are a very weak 
part of Git.

quoted

- git-annex (does not get the files in when cloning, pulling, checking 
  out; you need to do it manually)
- git-media (same as git-annex)

Yes, this is an option, but a bit klunky.

quoted

- boar (no, we do not want to use a VCS for binaries in addition to git)

I did not know about that.

quoted

- and a few more

So far the only good solution seems to be git-bigfiles but it's still
in development.

It has stalled, apparently, but I wanted to have a look at it anyway. Will 
let you know of my findings!

I think in many applications the "download-on-demand" approach which
git-annex takes is very important. (I don't know how far our
sparse/shallow supports this.) Also, their remote backends look
interesting. And no, I don't want Haskell as yet another language for
our code base.

Fedora handles big files (compressed tar balls) in git with a file
store, scripting (fedpkg) and tracking only a text file with hash values
("sources") in git; somehow a baby version of git-annex.

The symlink based approach of annex (big file is a symlink to the
"object store" which is indexed by blob content sha1) reminds me very
much of our notes trees and the way textconv-cache uses it. It feels as
if we already have all the pieces in place. (I don't think we need to
track big files' contents, only their hashes; this is fast for read-only
media, see annex' worm-backend.)

Another crazy idea would be to "git replace" big files by place-holders
(blob with the big file's sha1 as content) or rather the other way
round, but I haven't thought this through.

Michael

On Wed, Apr 6, 2011 at 5:25 AM, Johannes Schindelin
[off-list ref] wrote:

I understand. The problem in your case might not be too bad, after all.
The problem only arises when you have big files that are compressed. If
you check in multiple versions of an uncompressed .dll file, Git will
usually do a very good job at compressing them.

Except when they are very large; in that case git tends to OOM. But
just yesterday Junio posted a proposed patch to honour a max file size
for compression (search the archive for 'Git exhausts memory' and
'core.bigFileThreshold'.

So Pau might be in luck with current git + Junio's patch + enough RAM
on the workstations. Pau, I definitely suggest you try it out.

If it still consumes too much memory with the largest filest (ie: the
VM images you mention), the fedpkg approach (discussed in this thread)
is good too. It's a Python wrapper around git, which tracks a text
file listing the hashes of the large files, and fetches them (or
uploads them) via SCP. You only need to use it when dealing with the
large files -- most of the time you're just using git.

The fedpkg code is quite readable, and I've already "stolen" some of
its code for my local needs. Recommended.

cheers,


m
-- 
 martin.langhoff@gmail.com
 martin@laptop.org -- Software Architect - OLPC
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

Pau Garcia i Quiles [off-list ref] writes:

The usual answer to the "I need to put binaries in the repository"
question has been "no, you do not". Well, we do. We are in heavy
development now, therefore today's version may depend on a certain
version of a third-party shared library (DLL) which we only can get in
binary form, and tomorrow's version may depend on the next version of
that library, and you cannot mix today's source with yesterday's
third-party DLL. I. e. to be able to use the code from 7 days ago at
11.07 AM you need "git checkout" to "return" our source AND the
binaries we were using back then. This is something ClearCase manages
satisfactorily.

If it were me, I'd just store the huge binaries in some sort of separate
remote filesystem, and then store the remote-file-system _paths_ to them
in git (in a simple text file).

Then either use the build system or some sort of git filter to make sure
that the actual library was installed before building based on the path
read from the file in git.

[This would be a pain as a _general_ solution (for git), because it
involves coordination with a the remote file system, etc, but for an
organization like yours setting up a system for a specific product, it
should be fairly easy to set up and maintain -- and particularly so if
the main use is to store 3rd party library releases, as they're
typically not going to be something that anybody will want to checkin,
but rather installed by a small set of people.]

-Miles

-- 
Circus, n. A place where horses, ponies and elephants are permitted to see
men, women and children acting the fool.

Hi,

On Thu, 7 Apr 2011, Miles Bader wrote:

Pau Garcia i Quiles [off-list ref] writes:

quoted

The usual answer to the "I need to put binaries in the repository" 
question has been "no, you do not". Well, we do. We are in heavy 
development now, therefore today's version may depend on a certain 
version of a third-party shared library (DLL) which we only can get in 
binary form, and tomorrow's version may depend on the next version of 
that library, and you cannot mix today's source with yesterday's 
third-party DLL. I. e. to be able to use the code from 7 days ago at 
11.07 AM you need "git checkout" to "return" our source AND the 
binaries we were using back then. This is something ClearCase manages 
satisfactorily.

If it were me, I'd just store the huge binaries in some sort of separate 
remote filesystem, and then store the remote-file-system _paths_ to them 
in git (in a simple text file).

That fails for a number of reasons:

- it does not pass the 30,000-feet-high test

- integrity is not guaranteed (anybody can edit the files on the remote 
  file system, and nobody would realize that a "git checkout HEAD~2000" 
  ends up being something different from before)

- you would have to reinvent an efficient transfer (e.g. taking into 
  account all the data we have already)

- storage is no longer efficient, especially if you have multiple versions 
  of the same file.

- it is no longer decentralized anymore. Just think about yourself sitting 
  in the middle of antarctica, desperately needing to match a penguin 
  against a database of known penguins. You definitely want to have the 
  database local instead of leeching it down the non-existing wire all the 
  time. Likewise, if you and your group sit, say, on Viti Levu, and 
  develop software with people from New York, Texas, you definitely want
  a repository-in-the-middle, making it one person's duty to synchronize, 
  say, once per day.

I am sure you can think of more reasons.

Ciao,
Johannes