Re: Can git be tweaked to work cross-platform, on FAT32?

Thank you Martin and Johannes -

before proceeding by following either of your two suggestions, I
would like to make sure that there is no other way, so I would like to
ask:

1. I know, there is absolutely nothing in either of your answers that
   says that the route I conceived is not possible, but anyway: DO
   your answers imply that it is NOT possible (by "possible", I mean
   of course: possible without re-programming the source code ;-)) to
   compile git in a way that would enable it to work DIRECTLY with a
   local repository on FAT32 from both Windows and Linux (or, perhaps
   more simple: to work directly with a repository on FAT32 under
   Linux)?

   I'm repeating my question because it is impossible for me to tell
   whether this would involve a rather simple change (such as: don't
   use symlinks, and/or: don't care about user/group permissions), or
   whether the critical differences are rather an inherent
   characteristic of git, or perhaps even the libraries it uses.

   And: if this were possible, I would consider it a rather attractive
   idea, not only for me, hence I'd like to rule it out before
   dropping it altogether.


2. ad Martin: Ouf! I have no idea of how Samba works (well, yes: I
   thought it was some kind of server or client program permitting to
   connect to Windows servers or clients), hence I don't get a very
   clear idea of what you have in mind. I hope I have made it clear
   that I am using only one computer (double-boot, simply), and that
   everything is local. I am aware that Linux permits to do "virtual"
   networking with localhost, but how would your scheme work on a
   single machine? Can samba "pretend" there is a Windows network
   behind it, while in truth, it accesses a local directory (on
   FAT32)? And can git work directly in a repository it considers to
   be accessible over the network only?

   The interesting thing in this could be that one could use indeed
   the same repository under both OSes. If I got vaguely the right
   idea, that is.

3. ad Johannes: This does sound quite simple and straightforward. If I
   got it right, it would involve having one repository on a, say,
   ext2 partition to work with under Linux, and one on a FAT32
   partition to work with under Windows, and syncing the two after
   booting (fetching from FAT32) and before shutting down (pushing to
   FAT32) Linux.

   It is quite interesting, BTW, that git can /sync/ with a repository
   on FAT32 under Linux, but not work with it.


Thanks very much for both answers! I would still be curious to know
whether there is an answer to my first question, but yours should get
me working one way or other.

Regards, Florian

Johannes Schindelin writes:
 > On Mon, 18 Dec 2006, Stefano Spinucci wrote:
 > > On 12/17/06, Johannes Schindelin [off-list ref] wrote:
 > > > I am sure that there are more problems with it. BTW the problem stems from
 > > > Windows having _no_ equivalent to fork().
 > > 
 > > if you are sure that there are more problems with using MMAP, shouldn't 
 > > NO_MMAP be commented better in the Makefile and in INSTALL, pointing out 
 > > to use it always on cygwin ???
 > 
 > I argued for that a long time ago, but there were a couple "works-for-me"s 
 > and I just shut up.

It would be nice to have test cases documenting known failures. Then
people could decide whether these cases are important for their work,
since I don't think it's actually possible to loose data this way.

Personally I've been unsuccessful in trying to recreate failures, but
that is probably just an indication of my inability to use git
correctly:-)

On 12/17/06, Johannes Schindelin [off-list ref] wrote:

quoted

3. ad Johannes: This does sound quite simple and straightforward. If I
   got it right, it would involve having one repository on a, say,
   ext2 partition to work with under Linux, and one on a FAT32
   partition to work with under Windows, and syncing the two after
   booting (fetching from FAT32) and before shutting down (pushing to
   FAT32) Linux.

This is how I'd do it.

And I concur - I had only introduced Samba to the conversation because
I thought you were talking about several computers. If you are
dual-booting on one machine, I'd do as above.

Note that under windows you can use ext2 -- haven't used it, and don't
know how cygwin behaves with it, but it may be *just* what you need to
avoid case sensitivity problems and have symlink support.

    http://sourceforge.net/projects/ext2fsd

    http://www.fs-driver.org/

cheers,

Hi,

On Sun, 17 Dec 2006, Juergen Ruehle wrote:

Johannes claims that there are additional problems with mmap on cygwin, 
but it passes the complete test suite on NTFS, so it should be ok for 
most operations

I encountered a problem with git-log, where the fork() tried to 
reestablish a mmap()ed file, which unfortunately was renamed (since it was 
a .lock file[*1*]). This triggered the implementation of NO_MMAP.

I am sure that there are more problems with it. BTW the problem stems from 
Windows having _no_ equivalent to fork().

Notefoot 1: Isn't it funny that Windows cannot delete files which are 
still opened, but does not care when it is renamed?

Ciao,

Hi,

On Sun, 17 Dec 2006, Stefano Spinucci wrote:

I just tried to use git writing to my FAT32 formatted usb stick.

On windows XP, I compiled git with and without NO_D_TYPE_IN_DIRENT, but 
after the following actions I always got the error "fatal: Unable to 
write new index file" or "fatal: unable to create '.git/index': File 
exists":

Come to think of it, I probably never tried to actually _commit_ on 
FAT32... Sorry.

I will not be able to test this scenario until Tuesday, though. Sorry 
again!

Ciao,

On 12/17/06, Martin Langhoff [off-list ref] wrote:

Note that under windows you can use ext2 -- haven't used it, and don't
know how cygwin behaves with it, but it may be *just* what you need to
avoid case sensitivity problems and have symlink support.

    http://sourceforge.net/projects/ext2fsd

    http://www.fs-driver.org/

I've used both of these guys, but not the most recent versions. Both
were previously read-only, but now it appears both have write support
from looking at the release notes. Both were a bit heavy handed to get
setup, but a decently computer literate person could probably handle
it. Ext2IFS in particular looks dramatically improved from when I last
used it.

Both worked fine through Cygwin for simple usage. I never did anything
with serious disk usage on either of them, so I can't say how it
stacks up to FAT32 on Linux in terms of really hammering the disk.

A little more reading says ext2ifs can't handle permissions or symbolic links.

http://www.fs-driver.org/faq.html

I don't recall the details of ext2fsd, but I think symbolic links
worked. I'd like to look at this again, but I spend all my time in
Linux at home.

--

I just tried to use git writing to my FAT32 formatted usb stick.

On windows XP, I compiled git with and without NO_D_TYPE_IN_DIRENT,
but after the
following actions I always got the error "fatal: Unable to write new
index file" or
"fatal: unable to create '.git/index': File exists":

mkdir git-test-repo
cd git-test-repo
git-init-db
echo iii>test
git add test
git commit -m "test msg"
echo ooo>test1
git add test1 ***error***

converting the stick to NTFS I have no problems.

any hint ???

---

Stefano Spinucci writes:
 > I just tried to use git writing to my FAT32 formatted usb stick.
 > 
 > On windows XP, I compiled git with and without NO_D_TYPE_IN_DIRENT,
 > but after the
 > following actions I always got the error "fatal: Unable to write new
 > index file" or
 > "fatal: unable to create '.git/index': File exists":

Compile git with NO_MMAP. The problem is that the old index file is
mmaped while it is replaced. Cygwin supports this operation only on
NTFS. I don't know whether this is a bug in cygwin or a windows
limitation (I suspect the latter).

Unfortunately NO_MMAP makes git pretty slow if the pack files get
larger than about 10-20 MB.

Johannes claims that there are additional problems with mmap on
cygwin, but it passes the complete test suite on NTFS, so it should be
ok for most operations

IIRC it has been suggested on the list that mmaping the index isn't
that important for git performance and it could be replaced by simply
reading the index file into memory on access.

It would be nice to have mmap work on non-NTFS partitions since my
tests suggest that FAT32 is up to twice as fast (in the hot cache
case). (My test version simply uses an unlink before the rename in
lockfile.c to hack around the cygwin rename problem.)

On 12/17/06, Johannes Schindelin [off-list ref] wrote:

...
I am sure that there are more problems with it. BTW the problem stems from
Windows having _no_ equivalent to fork().
...

I recompiled git with NO_MMAP and used it successfully, then I added a
note in the wiki
page http://git.or.cz/gitwiki/WindowsInstall on how to compile git to
work on a FAT32
filesystem.

Now, two questions...

if you are sure that there are more problems with using MMAP, shouldn't NO_MMAP
be commented better in the Makefile and in INSTALL, pointing out to
use it always on
cygwin ???

finally on cygwin there is a precompiled git package (v 1.4.4.1);
however, I found if it was
compiled *with* MMAP support... should I write to the cygwin-apps list
asking for a
different compilation (with NO_MMAP), or for the majority of the users
the standard
compilation is a better choice ???

bye

---

Hi,

On Mon, 18 Dec 2006, Stefano Spinucci wrote:

On 12/17/06, Johannes Schindelin [off-list ref] wrote:

quoted

...
I am sure that there are more problems with it. BTW the problem stems from
Windows having _no_ equivalent to fork().
...

if you are sure that there are more problems with using MMAP, shouldn't 
NO_MMAP be commented better in the Makefile and in INSTALL, pointing out 
to use it always on cygwin ???

I argued for that a long time ago, but there were a couple "works-for-me"s 
and I just shut up.

It could also be the case that for most operations, git became safe over 
time. For example, my biggest problem was the fork()ed diff in git-log, 
which was substituted by a non-forking inbuilt diff.

should I write to the cygwin-apps list asking for a different 
compilation (with NO_MMAP), or for the majority of the users the 
standard compilation is a better choice ???

Given that most Windows setups nowadays run on NTFS, I doubt that this is 
necessary. I use NO_MMAP myself on Windows, just in case, though.

Ciao,
Dscho

Hi,

On Sat, 16 Dec 2006, Florian v. Savigny wrote:

DO your answers imply that it is NOT possible (by "possible", I mean of 
course: possible without re-programming the source code ;-)) to compile 
git in a way that would enable it to work DIRECTLY with a local 
repository on FAT32 from both Windows and Linux (or, perhaps more 
simple: to work directly with a repository on FAT32 under Linux)?

My answer was not so clear...

Question: Is it possible to use git on a FAT32 filesystem?

Answer: Yes.

Question: Is it possible to access such a repository from both Linux and 
Windows?

Answer: Yes.

Question: Is there anything that could cause problems?

Answer: Yes. The working directory is a problem.

	- Symlinks for example. Don't use them.
	- Umlauts: don't use them.
	- The index has to be updated, since the stat information is 
	  unlikely to match. A simple "git status" should suffice, though.
	- Permissions: the permission handling of Linux and Windows are
	  quite different. If you initialize the repository with Cygwin,
	  it automatically sets core.filemode=false, if you initalize(d) 
	  with Linux, you have to adjust manually.

Question: Are there other shortcomings to this approach?

Answer: Definitely. Most are only annoying (like the performance of FAT32, 
which literally sucks), but some are different: for example, if you 
compile things in the working directory, you will be unhappy.

3. ad Johannes: This does sound quite simple and straightforward. If I
   got it right, it would involve having one repository on a, say,
   ext2 partition to work with under Linux, and one on a FAT32
   partition to work with under Windows, and syncing the two after
   booting (fetching from FAT32) and before shutting down (pushing to
   FAT32) Linux.

This is how I'd do it.

   It is quite interesting, BTW, that git can /sync/ with a repository
   on FAT32 under Linux, but not work with it.

You can. But I wouldn't. I like to keep separations clean and obvious.

Ciao,