Using GIT to store /etc (Or: How to make GIT store all file permission bits)

I've recently become somewhat interested in the idea of using GIT to  
store the contents of various folders in /etc.  However after a bit  
of playing with this, I discovered that GIT doesn't actually preserve  
all permission bits since that would cause problems with the more  
traditional software development model.  I'm curious if anyone has  
done this before; and if so, how they went about handling the  
permissions and ownership issues.

I spent a little time looking over how GIT stores and compares  
permission bits; trying to figure out if it's possible to patch in a  
new configuration variable or two; say "preserve_all_perms" and  
"preserve_owner", or maybe even "save_acls".  It looks like standard  
permission preservation is fairly basic; you would just need to patch  
a few routines which alter the permissions read in from disk or  
compare them with ones from the database.  On the other hand, it  
would appear that preserving ownership or full POSIX ACLs might be a  
bit of a challenge.

Thanks for your insight and advice!

Cheers,
Kyle Moffett

I want to have a tripwire-like system checking the files to make sure that they 
haven't changed unexpectedly. the program I'm looking at notices inode as well 
as timestamp and content changed.

when you checkout a file from git will it re-write/overwrite a file that hasn't 
changed or will it realize there is no change and leave it as-is?

does this answer change if there is a trigger on checkout (to change permissions 
or otherwise manipulate the file)?

David Lang

David Lang [off-list ref] wrote:

I want to have a tripwire-like system checking the files to make sure that 
they haven't changed unexpectedly. the program I'm looking at notices inode 
as well as timestamp and content changed.

when you checkout a file from git will it re-write/overwrite a file that 
hasn't changed or will it realize there is no change and leave it as-is?

If the stat data is current it will leave it as-is.  You can force
the index to refresh with `git update-index --refresh` or by running
git status.
 

does this answer change if there is a trigger on checkout (to change 
permissions or otherwise manipulate the file)?

Only if the trigger does something in addition, like force overwrite
files.  But we don't have a checkout trigger.  So there's no trigger.

-- 
Shawn.

On Tue, 9 Jan 2007, Shawn O. Pearce wrote:

David Lang [off-list ref] wrote:

quoted

I want to have a tripwire-like system checking the files to make sure that
they haven't changed unexpectedly. the program I'm looking at notices inode
as well as timestamp and content changed.

when you checkout a file from git will it re-write/overwrite a file that
hasn't changed or will it realize there is no change and leave it as-is?

If the stat data is current it will leave it as-is.  You can force
the index to refresh with `git update-index --refresh` or by running
git status.

I was looking at checkout, not checkin so I'm not understanding how the index is 
involved here.

quoted

does this answer change if there is a trigger on checkout (to change
permissions or otherwise manipulate the file)?

Only if the trigger does something in addition, like force overwrite
files.  But we don't have a checkout trigger.  So there's no trigger.

we don't have a checkout trigger? I thought that what Linus had suggested for 
permissions was to have a script triggered on checkin that stored the 
permissions of the files, and a script triggered on checkout that set the 
permissions from the stored file.

if there isn't a checkout trigger how would the permissions ever get set?

in my particular case I'd like to have the checkin run a script that produces a 
'generic' version of each file, and the checkout run a script that converts the 
generic version into the host specific version. I already have a script that 
does this work (and (ab)uses ssh to propogate the generic version to other hosts 
and create the host specific versions there), but I was interested in useing git 
to add better version control to the generic versions of the files (I currently 
use RCS on each box to version control the host specific versions)

David Lang

David Lang [off-list ref] wrote:

On Tue, 9 Jan 2007, Shawn O. Pearce wrote:

quoted

If the stat data is current it will leave it as-is.  You can force
the index to refresh with `git update-index --refresh` or by running
git status.

I was looking at checkout, not checkin so I'm not understanding how the 
index is involved here.

During checkout we use the index to help us decide if a file needs
to be updated with new content or can be left as-is.  Its a cache of
what version each file is at, and its based on the file stat data
(dev, inode, modification date, etc.) to tell us if the file has
been modified or was last created by Git.  If Git was the one that
last modified the file and the version stored in the index matches
the version needed during the checkout, the file is left alone.
But if anything differs then the file gets overwritten.
 

quoted

quoted

does this answer change if there is a trigger on checkout (to change
permissions or otherwise manipulate the file)?

Only if the trigger does something in addition, like force overwrite
files.  But we don't have a checkout trigger.  So there's no trigger.

we don't have a checkout trigger?

No.

I thought that what Linus had suggested 
for permissions was to have a script triggered on checkin that stored the 
permissions of the files, and a script triggered on checkout that set the 
permissions from the stored file.

Yes.  It is what he suggested.

if there isn't a checkout trigger how would the permissions ever get set?

Someone needs to implement support for a post-checkout trigger.  _Then_
a checkout trigger could perform this action.

in my particular case I'd like to have the checkin run a script that 
produces a 'generic' version of each file,

You may be able to do that in the pre-commit hook by updating the index

-- 
Shawn.

Hi,

On Tue, 12 Dec 2006, Steven Grimm wrote:

If there were a shallowize command, you could do something like "git 
pull; git shallowize --depth 1" to track the latest revision without 
keeping the history locally.

Almost!

$ git pull --depth 1

Though it needs a server _and_ a client supporting shallow clones, which 
support is brewed in "next" right now.

Ciao,
Dscho

On Monday 11 December 2006 11:39, Andreas Ericsson wrote:

quoted

Import/export scripts literally require wrapping every single GIT 
command with a script that changes directory a few times, reads from a 
different checked-out tree, and permutes some extended-attribute data 
slightly before storing it in the underlying GIT tree.  Even without 
adding any new functionality whatsoever that doubles the amount of code 
just for finding your repository and checking command-line arguments, 
and that's a crazy trade-off to make in any situation.

GIT_DIR=/some/where/else/.git git log -p

Doing this everytime you want to run a git command *is* a lot of time
wasted for typing.

The .gitlink proposal would come in handy here: you have a simple
file instead of .git/, which links to the real repository.

This discussion reminds me of a use of git I've had in the back of my 
head to try out for a while. Right now I'm doing my local snapshot 
backups using the rsync-with-hard-links scheme 
(http://www.mikerubel.org/computers/rsync_snapshots/ if you're not 
familiar with it). This is nice in that the contents of files that don't 
change are only stored once on the backup disk. But it is less than 
optimal in that a file that changes even a little bit is stored from 
scratch.

What would be great for this would be to store each day's backup as a 
git revision; with a periodic repack, this would be much more 
space-efficient than the rsync hard links.

The problem is that while that would give me a very efficient backup 
scheme, the repository would still grow over time. In rsync land, I 
solve the disk space issue by keeping two weeks' worth of daily 
snapshots, then six months' worth of weekly snapshots, then two years' 
worth of monthly snapshots; files that change daily have a constant 
number of revisions stored in my backups, and older files drop off the 
backup disk as they age.

Given that there's no way (or is there?) to delete revisions from the 
*beginning* of a git revision history, right now it seems like the only 
approach that comes close is to give up on the "daily then weekly then 
monthly" thing -- probably fine given the space savings of delta 
compression -- and periodically make shallow clones of the backup 
repository that fetch all but the first N revisions; once a shallow 
clone is made, the original gets deleted and the clone is the new backup 
repo.

But it would sure be more efficient to be able to "shallow-ize" an 
existing repository. That would be useful for things other than backups, 
too, e.g. the recent request for some way to track just the current 
version of the kernel code rather than its revision history. If there 
were a shallowize command, you could do something like "git pull; git 
shallowize --depth 1" to track the latest revision without keeping the 
history locally.

Anyone think that sounds like an interesting thing to explore?

-Steve

On Monday 11 December 2006 14:33, Johannes Schindelin wrote:

On Mon, 11 Dec 2006, Josef Weidendorfer wrote:

quoted

On Monday 11 December 2006 11:39, Andreas Ericsson wrote:

quoted

quoted

Import/export scripts literally require wrapping every single GIT 
command with a script that changes directory a few times, reads from a 
different checked-out tree, and permutes some extended-attribute data 
slightly before storing it in the underlying GIT tree.  Even without 
adding any new functionality whatsoever that doubles the amount of code 
just for finding your repository and checking command-line arguments, 
and that's a crazy trade-off to make in any situation.

GIT_DIR=/some/where/else/.git git log -p

Doing this everytime you want to run a git command *is* a lot of time
wasted for typing.

The .gitlink proposal would come in handy here: you have a simple
file instead of .git/, which links to the real repository.

I beg your pardon; I'm just joining in. Why is a symbolic link for .git 
inacceptable?

You are totally right.

The .gitlink thing is tailored to allow submodule support later. It includes
some smart searching for the git repository to allow moving the checkout in
some limits without breaking the link to the repository.

Aside from this, the proposal is more flexible in that you can specify not
only GIT_DIR (or the GIT_DIR_HINT to trigger smart search), but also
GIT_INDEX_FILE and GIT_HEAD_FILE, which allows different checkouts
(with different index state and HEAD) for the same repo easily.

Which is not needed in this case.
So, sorry for the noise ;-)

Kyle Moffett wrote:

On Dec 10, 2006, at 13:26:32, Jakub Narebski wrote:

quoted

The idea is to not store /etc in git directly, but use import/export 
scripts, which for example saves permissions and ownership in some 
file also tracked by git on import, and restores correct permissions 
on export. That is what I remember from this discussion. This of 
course means that you would have to write your own porcelain...

What about mentioned in other email IsiSetup?

The real problem I have with that is you literally have to duplicate all 
sorts of functionality.  I want to run "foo-status" in /etc and get 
something useful, but if /etc is not a git directory in and of itself 
then you have to duplicate most of "git-status" anyways.

Make /etc/.git a symlink to where you store your repo and go to the 
other directory when you want to *restore* configuration. The only "own 
porcelain" you need to write is a simple program that understands "save" 
and "restore" (or some such) and tucks away the meta-data in a file 
somewhere inside the git tree. If you make it in the format

octal-mode path/to/file

you can even get decently human-readable permission diffs, which will 
most likely be prettier and easier to read than anything git currently has.

GIT already has _some_ idea about file permissions, it just discards 
most of the data before writing to disk.   Of course, adding POSIX ACLs
and user-extended-attributes requires a new data format, but those are 
very similar to filesystem permissions; they differ only in amount of 
data stored, not in purpose.

The amount of data stored is the issue here. The current implementation 
(which works just fine and does The Right Thing(tm) for code-repos) only 
stores what it has to and uses the spare bits to do other things.

Import/export scripts literally require wrapping every single GIT 
command with a script that changes directory a few times, reads from a 
different checked-out tree, and permutes some extended-attribute data 
slightly before storing it in the underlying GIT tree.  Even without 
adding any new functionality whatsoever that doubles the amount of code 
just for finding your repository and checking command-line arguments, 
and that's a crazy trade-off to make in any situation.

GIT_DIR=/some/where/else/.git git log -p

Why would you want to read from a different checked-out tree? 
Non-committed data is "changes", committed data is "HEAD" (or 
commit-ish) and marked data is "index". I see no reason what so ever for 
a second checked-out tree.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se

Kyle Moffett wrote:

On Dec 10, 2006, at 10:30:00, Jakub Narebski wrote:

quoted

Jeff Garzik wrote:

quoted

I actively use git to version, store and distribute an exim mail  
configuration across six servers.  So far my solution has been a  
'fix perms' script, or using the file perm checking capabilities  
of cfengine.

Fix perms' script used on a checkout hook is a best idea I think.

Hmm, unfortunately that has problems with security-related race  
conditions when used directly for /etc.  Think about what happens  
with "/etc/shadow" in that case, for example.  (/etc/.git is of  
course 0700)  I'm sure there are others where non-root daemons get  
unhappy when they get an inotify event and their config files have  
suddenly become root:root:0600.  I also want to be able to "cd /etc  
&& git status" to see what changed after running "apt-get update" or  
maybe fiddling in SWAT or webmin, so a makefile which installs into / 
etc won't quite solve it either.  It would also be nice to see when  
things change the permissions on files in /etc, or even bind-mount an  
append-only volume over /etc/.git/objects to provide additional data  
security.

The idea is to not store /etc in git directly, but use import/export
scripts, which for example saves permissions and ownership in some
file also tracked by git on import, and restores correct permissions
on export. That is what I remember from this discussion. This of course
means that you would have to write your own porcelain...

What about mentioned in other email IsiSetup?

-- 
Jakub Narebski
Warsaw, Poland

Kyle Moffett wrote:

On Dec 10, 2006, at 10:30:00, Jakub Narebski wrote:

quoted

Jeff Garzik wrote:

quoted

I actively use git to version, store and distribute an exim mail  
configuration across six servers.  So far my solution has been a  
'fix perms' script, or using the file perm checking capabilities  
of cfengine.

Fix perms' script used on a checkout hook is a best idea I think.

Hmm, unfortunately that has problems with security-related race  
conditions when used directly for /etc.  Think about what happens  
with "/etc/shadow" in that case, for example.  (/etc/.git is of  
course 0700)  I'm sure there are others where non-root daemons get  
unhappy when they get an inotify event and their config files have  
suddenly become root:root:0600.  I also want to be able to "cd /etc  
&& git status" to see what changed after running "apt-get update" or  
maybe fiddling in SWAT or webmin, so a makefile which installs into / 
etc won't quite solve it either.  It would also be nice to see when  
things change the permissions on files in /etc, or even bind-mount an  
append-only volume over /etc/.git/objects to provide additional data  
security.

The idea is to not store /etc in git directly, but use import/export
scripts, which for example saves permissions and ownership in some
file also tracked by git on import, and restores correct permissions
on export. That is what I remember from this discussion. This of course
means that you would have to write your own porcelain...

What about mentioned in other email IsiSetup?
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

On 12/10/06, Kyle Moffett [off-list ref] wrote:

I've recently become somewhat interested in the idea of using GIT to
store the contents of various folders in /etc.  However after a bit
of playing with this, I discovered that GIT doesn't actually preserve
all permission bits since that would cause problems with the more
traditional software development model.  I'm curious if anyone has
done this before; and if so, how they went about handling the
permissions and ownership issues.

I keep the files I want to track in a separate folder that I track
with Git and use a Makefile for updating /etc.  I basically have a
rule for checking for differences between the tracked folder and /etc
and a rule for installing changed files (with the correct
permissions).  It works, but it does require some "Makefile magic" to
work right (or the way /I/ want it anyway).

On Tue, 12 Dec 2006, Kyle Moffett wrote:

Hmm, ok.  It would seem to be a reasonable requirement that if you want to
change any of the "preserve_*_attributes" config options you need to blow away
and recreate your index, no?  I would probably change the underlying index
format pretty completely and stick a new version tag inside it.

You should be able to promote an insufficient-version index to a 
new-version index that's needs to be refreshed for every entry. (And then 
update-index would take care of the necessary rewrite-everything in the 
normal way). But I suspect that the right thing is to require that the 
repository be created with a "commits-include-directories-not-trees" flag, 
and this means that you always use the extra-detailed index, and the 
options only affect what information is filtered out in transit between 
the directory object and the index. Having more information in the index 
is merely a potential waste of space, not a correctness issue (we have 
extra information for trees in the index now, remember); it just means 
that there are more things that will cause git to reread the file, rather 
than declaring it unchanged with a stat().

For that matter, it may be best for the directory objects to record what 
information in them is real, and keep the "what's content" mask in the 
index as well. If it changes over the history of a repository, you want to 
correctly interpret the historical commits.

Ok, seems straightforward enough.  One other thing that crossed my mind was
figuring out how to handle hardlinks.  The simplest solution would be to add
an extra layer of indirection between the "file inode" and the "file data".
Instead of your directory pointing to a "file-data" blob and "file-attributes"
object, it would point to an "file-inode" object with embedded attribute data
and a pointer to the file contents blob.

I remember reading some discussions from the early days of GIT about how that
was considered and discarded because the extra overhead wouldn't give any real
tangible benefit.  On the other hand for something like /etc the added
benefits of tracking extended attributes and hardlinks might outweigh the cost
of a bunch of extra objects in the database.  A bit of care with the
construction of the index file should make it sufficiently efficient for
day-to-day usage.

I was thinking this could be internal to the directory object, but you 
probably want to support hardlinks shared between dentries in different 
directory objects, so you're probably right that this makes sense. 

Alternatively, you could use a single "directory" object for the whole 
state (including subdirectories), making hardlinks out of the object 
clearly impossible, or you could use some scheme for sharing 
sub-"directory" objects that would imply that hardlinks are within an 
object (the hard part here is finding things when their locations aren't 
predictable by name).

	-Daniel

On Tuesday 2006 December 12 13:49, Kyle Moffett wrote:

Hmm, ok.  It would seem to be a reasonable requirement that if you
want to change any of the "preserve_*_attributes" config options you
need to blow away and recreate your index, no?  I would probably
change the underlying index format pretty completely and stick a new
version tag inside it.

I wonder if git's skill at managing content is the answer?  Rather than mess 
around with git's internals, the index, or the object database; how about 
simply having a pre-commit script that writes out a file that looks like:

-rw-r--r--  andyp andyp CHANGES
-rw-r--r--  andyp andyp COPYING
-rw-rw-r--  andyp andyp CREDITS
-rw-r--r--  andyp andyp Configure
-rw-rw-r--  andyp andyp Makefile
-rw-r--r--  andyp andyp README

If /that/ file were stored in the repository and you had a script that could 
read that file and apply the permissions after a checkout you'd have what you 
want.

If the permissions of a file changed but the content didn't, then 
this ".gitpermissions" file would have changed content but the file itself 
would remain the same.  If the content changed but not the permissions 
then ".gitpermissions" would be untouched.

Assuming that you're allowed to mess with the index in pre-commit (I haven't 
checked), one half of it can be automatic.  I suppose you could also plead 
for a post-checkout hook to apply those permissions and the whole lot would 
be transparent.



Andy
-- 
Dr Andy Parkins, M Eng (hons), MIEE

On Dec 10, 2006, at 13:26:32, Jakub Narebski wrote:

The idea is to not store /etc in git directly, but use import/ 
export scripts, which for example saves permissions and ownership  
in some file also tracked by git on import, and restores correct  
permissions on export. That is what I remember from this  
discussion. This of course means that you would have to write your  
own porcelain...

What about mentioned in other email IsiSetup?

The real problem I have with that is you literally have to duplicate  
all sorts of functionality.  I want to run "foo-status" in /etc and  
get something useful, but if /etc is not a git directory in and of  
itself then you have to duplicate most of "git-status" anyways.  And  
the same applies to all the other commands.  From what I can see of  
IsiSetup the tools for checking out, merging, modifying, cloning, etc  
are all much more limited and immature than the ones available  
through GIT/cogito, and I would be loathe to discard all that extra  
functionality and duplicate a few thousand lines of code in the name  
of "concept purity".

GIT already has _some_ idea about file permissions, it just discards  
most of the data before writing to disk.  Of course, adding POSIX  
ACLs and user-extended-attributes requires a new data format, but  
those are very similar to filesystem permissions; they differ only in  
amount of data stored, not in purpose.

Import/export scripts literally require wrapping every single GIT  
command with a script that changes directory a few times, reads from  
a different checked-out tree, and permutes some extended-attribute  
data slightly before storing it in the underlying GIT tree.  Even  
without adding any new functionality whatsoever that doubles the  
amount of code just for finding your repository and checking command- 
line arguments, and that's a crazy trade-off to make in any situation.

Cheers,
Kyle Moffett

On Dec 11, 2006, at 22:45:25, Daniel Barkalow wrote:

The first thing you'd want to do is correct the fact that the index  
doesn't keep full permissions. We decided long ago that we don't  
want to track more than 0100, but we're discarding the rest between  
the filesystem and the index, rather than between the index and the  
tree. (This is weird of us, since we keep gid and uid in the index,  
as changedness heuristics, but don't keep permissions; of course,  
we'd have to apply umask to the index when we check it out to sync  
what we expect to be there with what has actually been created.)

I think that would be the only change needed to the index and index/ 
working directory connection, although it might be necessary to  
support longer values for uid/gid/etc, since they'd be important  
data now.

Hmm, ok.  It would seem to be a reasonable requirement that if you  
want to change any of the "preserve_*_attributes" config options you  
need to blow away and recreate your index, no?  I would probably  
change the underlying index format pretty completely and stick a new  
version tag inside it.

Note that git only stores content, not incidental information. But  
a lot of information which is incidental in a source tree is  
content in /etc. This implies that /etc and working/linux-2.6 are  
fundamentally different sorts of things, because different aspects  
of them are content.

Ahh, I hadn't thought of it that way before but that makes a lot of  
sense.  Thanks!

I'd suggest a new object type for a directory with permissions,  
ACLs, and so forth. It should probably use symbolic owner and  
group, too. My guess is that you'll want to use "commit"s, the new  
object type, and "blob"s. Everything that uses trees would need to  
have a version that uses the new type. But I think that you  
generally want different behavior anyway, so that's not a major issue.

Ok, seems straightforward enough.  One other thing that crossed my  
mind was figuring out how to handle hardlinks.  The simplest solution  
would be to add an extra layer of indirection between the "file  
inode" and the "file data".  Instead of your directory pointing to a  
"file-data" blob and "file-attributes" object, it would point to an  
"file-inode" object with embedded attribute data and a pointer to the  
file contents blob.

I remember reading some discussions from the early days of GIT about  
how that was considered and discarded because the extra overhead  
wouldn't give any real tangible benefit.  On the other hand for  
something like /etc the added benefits of tracking extended  
attributes and hardlinks might outweigh the cost of a bunch of extra  
objects in the database.  A bit of care with the construction of the  
index file should make it sufficiently efficient for day-to-day usage.

If you're interested in some random musings about using GIT concepts  
to version whole filesystems (think checkpointing your disk drive and  
instantly restoring when you screw up), read on below, otherwise  
don't bother.

Cheers,
Kyle Moffett

<Random Tangential Off-the-Wall Thought Experiment>

NOTE: This probably belongs in it's own thread but it's such a  
random, undeveloped, and off-the-wall concept that I threw it in here  
just for kicks.

Combining extensions like those described above with something like  
the Ext3 block-allocation, inode-management and journalling code to  
produce a "versioned filesystem".  With the exponential growth of  
storage density over the last several years we've gotten to the point  
where we can many many hours of extremely realistic video and audio  
on your average small-computer drive.  Versioning your home  
directory, or even your entire computer, even with fairly steady  
modifications to multimedia files, installation of software programs,  
etc, doesn't seem like such an impossible undertaking anymore.

One predefined inode would contain a list of tags/heads and their  
current hashes.  Mount the filesystem with a "tag=$TAG" option to  
specify the initial tree object used for the root directory (with  
syscalls to navigate the history).  Allocate an inode per-mount to  
represent any changes from the last commit.

For efficiency purposes (no need to revision the entire system when I  
commit a change in my home directory) add a "subtree" object type  
which can specify either a particular hash or a symbolic tag/head  
name as a pseudo sub-mountpoint.  Trap traversal of the sub- 
mountpoint node to mount the filesystem with "tag=$SUBTAG" on the sub- 
mountpoint, expiring it some time after the last traversal.

The only remaining issue would be properly navigating through the  
history, preserving or discarding changes.  Since the kernel could  
easily manage copy-on-write semantics for underlying disk blocks you  
wouldn't need a separate "working copy" except where it's modified  
from the original, and discarding changes is as simple as unlinking  
any files referenced by the per-mount delta inode.

Committing changes would get tricky, you would need to hot-remap  
memory-mapped pages read-only while you checksum and store them.  The  
next write attempt would then separate the page from the freshly- 
committed on-disk version.  Would need a mechanism for applications  
to "trap" the commit so they could make databases consistent, with  
the ability for root or the mountpoint owner to commit without  
waiting for synchronization.  Only needs to synchronize files  
belonging to the new commit.  Merges would be managed from userspace,  
as long as there is a way to browse through objects by hash given  
sufficient permissions.

Make sure it's really easy to make a new atomic commit and/or reset  
to a known state every time the computer is rebooted (whether soft- 
rebooted or via crash/powerkill).  With journalling and the write- 
once nature of GIT it would be trivial to never require an fsck run.   
Also needs a way to move data between filesystems.  Makes LVM largely  
irrelevant; it doesn't matter how many disks you have if they're all  
treated as a shared storage pool for your GITfs data.  Make sure it's  
possible to archive data onto slower disks/media and purge older  
commits from the archive (missing parent commit references are  
tolerable in many situations).  Needs a way to notice hash collisions  
and take action to avoid them.

</Random Tangential Off-the-Wall Thought Experiment>

Cheers,
Kyle Moffett

Johannes Schindelin wrote:

$ git pull --depth 1

Though it needs a server _and_ a client supporting shallow clones, which 
support is brewed in "next" right now.
  

Will that actually discard old revisions that are already stored locally?

-Steve

On Dec 10, 2006, at 10:30:00, Jakub Narebski wrote:

Jeff Garzik wrote:

quoted

I actively use git to version, store and distribute an exim mail  
configuration across six servers.  So far my solution has been a  
'fix perms' script, or using the file perm checking capabilities  
of cfengine.

Fix perms' script used on a checkout hook is a best idea I think.

Hmm, unfortunately that has problems with security-related race  
conditions when used directly for /etc.  Think about what happens  
with "/etc/shadow" in that case, for example.  (/etc/.git is of  
course 0700)  I'm sure there are others where non-root daemons get  
unhappy when they get an inotify event and their config files have  
suddenly become root:root:0600.  I also want to be able to "cd /etc  
&& git status" to see what changed after running "apt-get update" or  
maybe fiddling in SWAT or webmin, so a makefile which installs into / 
etc won't quite solve it either.  It would also be nice to see when  
things change the permissions on files in /etc, or even bind-mount an  
append-only volume over /etc/.git/objects to provide additional data  
security.

quoted

But it would be a lot better if git natively cared about ownership  
and  permissions (presumably via an option).

There is currently no place for ownership and extended attributes  
in the tree object; and even full POSIX permissions might be  
challenge because for example currently unused 'is socket'  
permission bit is used for experimental commit-in-tree submodule  
support.

What about doing something crazy like "is socket" && "is directory"  
&& "is symlink"?  Or something else that old GIT versions would  
ignore and new GIT versions could do something useful with?  Perhaps  
like I mentioned in an earlier email, the new data could be stored as  
part of a modified "file" object.  Alternatively could a directory  
have a file named with an empty string with bogus mode bits which  
points to an extended-attributes-tree object?

And given Linus stance that git is "content tracker"...

Extended attributes are content too!  This includes things like  
icons, security labels (Think unclassified/confidential/secret/top- 
secret/etc), ACLs, summaries, and other metadata.  Content tracker  
purists could also just ignore the new default-off config options and  
be perfectly happy with status-quo. :-D

In the loooong thread "VCS comparison table" there was some talk  
about using git (or any SCM) to manage /etc. Check out:

 * Message-ID: [off-list ref]
   http://permalink.gmane.org/gmane.comp.version-control.git/29765
 * Message-ID: [off-list ref]
   http://marc.theaimsgroup.com/? 
i=[off-list ref]

(and other messages in this subthread).

I have, and while it's interesting material that thread produced no  
real patches :-D.  I'd like to introduce some new config options to  
control the new code: "preserve_full_perms", "preserve_posix_acls",  
"preserve_security_labels", and "preserve_user_xattrs" which default  
to false but when set modify GIT's behavior to store, retrieve, and  
compare additional data.

If you have any suggestions on how to store the data such that old  
GIT ignores it I'm all ears :-D.

Cheers,
Kyle Moffett

Another option is to have a process that stores your configs in git, and 
script an export from git to rpm|deb.  Packaging systems make it even 
easier to go between config versions.

	Jeff

Hi,

On Mon, 11 Dec 2006, Josef Weidendorfer wrote:

On Monday 11 December 2006 11:39, Andreas Ericsson wrote:

quoted

quoted

Import/export scripts literally require wrapping every single GIT 
command with a script that changes directory a few times, reads from a 
different checked-out tree, and permutes some extended-attribute data 
slightly before storing it in the underlying GIT tree.  Even without 
adding any new functionality whatsoever that doubles the amount of code 
just for finding your repository and checking command-line arguments, 
and that's a crazy trade-off to make in any situation.

GIT_DIR=/some/where/else/.git git log -p

Doing this everytime you want to run a git command *is* a lot of time
wasted for typing.

The .gitlink proposal would come in handy here: you have a simple
file instead of .git/, which links to the real repository.

I beg your pardon; I'm just joining in. Why is a symbolic link for .git 
inacceptable?

Ciao,
Dscho

On 12/10/06, Kyle Moffett [off-list ref] wrote:

I've recently become somewhat interested in the idea of using GIT to
store the contents of various folders in /etc.  However after a bit
of playing with this, I discovered that GIT doesn't actually preserve
all permission bits since that would cause problems with the more
traditional software development model.  I'm curious if anyone has
done this before; and if so, how they went about handling the
permissions and ownership issues.

I spent a little time looking over how GIT stores and compares
permission bits; trying to figure out if it's possible to patch in a
new configuration variable or two; say "preserve_all_perms" and
"preserve_owner", or maybe even "save_acls".  It looks like standard
permission preservation is fairly basic; you would just need to patch
a few routines which alter the permissions read in from disk or
compare them with ones from the database.  On the other hand, it
would appear that preserving ownership or full POSIX ACLs might be a
bit of a challenge.

Thanks for your insight and advice!

I have not used it, but you could try:

http://www.isisetup.ch/

that uses git as a backend.

Jeff Garzik wrote:

Kyle Moffett wrote:

quoted

I've recently become somewhat interested in the idea of using GIT to 
store the contents of various folders in /etc.  However after a bit of 
playing with this, I discovered that GIT doesn't actually preserve all 
permission bits since that would cause problems with the more 
traditional software development model.  I'm curious if anyone has done 
this before; and if so, how they went about handling the permissions and 
ownership issues.

I spent a little time looking over how GIT stores and compares 
permission bits; trying to figure out if it's possible to patch in a new 
configuration variable or two; say "preserve_all_perms" and 
"preserve_owner", or maybe even "save_acls".  It looks like standard 
permission preservation is fairly basic; you would just need to patch a 
few routines which alter the permissions read in from disk or compare 
them with ones from the database.  On the other hand, it would appear 
that preserving ownership or full POSIX ACLs might be a bit of a challenge.

It's a great idea, something I would like to do, and something I've 
suggested before.  You could dig through the mailing list archives, if 
you're motivated.

I actively use git to version, store and distribute an exim mail 
configuration across six servers.  So far my solution has been a 'fix 
perms' script, or using the file perm checking capabilities of cfengine.

Fix perms' script used on a checkout hook is a best idea I think.
 

But it would be a lot better if git natively cared about ownership and 
permissions (presumably via an option).

There is currently no place for ownership and extended attributes in
the tree object; and even full POSIX permissions might be challenge
because for example currently unused 'is socket' permission bit is
used for experimental commit-in-tree submodule support. And given Linus
stance that git is "content tracker"...

In the loooong thread "VCS comparison table" there was some talk
about using git (or any SCM) to manage /etc. Check out:

 * Message-ID: [off-list ref]
   http://permalink.gmane.org/gmane.comp.version-control.git/29765
 * Message-ID: [off-list ref]
   http://marc.theaimsgroup.com/?i=[off-list ref]

(and other messages in this subthread).
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Steven,

I've been thinking myself of writing a pdumpfs lookalike that uses git
internally. Sounds you you've got one already ;-)

In terms of getting rid of old history, have you considered moving a
graft point "forward" in time, and running git-repack -a -d? With your
history being (mostly?) linear this could be a workable scheme, but I
don't have much practice with using grafts.

cheers,

Hi,

On Tue, 12 Dec 2006, Steven Grimm wrote:

Johannes Schindelin wrote:

quoted

$ git pull --depth 1

Though it needs a server _and_ a client supporting shallow clones, 
which support is brewed in "next" right now.

Will that actually discard old revisions that are already stored 
locally?

No. A pull should _never_ lose anything from the repository. However, if 
some objects become no-longer reachable (and at the moment it looks like 
we cut of history, even if we should not need to), they can be pruned from 
the repo.

Hth,
Dscho

So, I've been making little repositories for appropriately related
stuff.  For example, I have a repository for my ~/.bashrc,
~/.bash_profile, ~/.bash_completions/*, and such.

I recall Linus's post in the "VCS Comparison Table" thread, and after
thinking about it, I decided the best thing to do would be to have a
couple extra files tracked in the repository, alongside other data.

I use a backup shell script to copy things from my system to the
repository, and then I run getfacl on it all to write out all the
details to a 'facl' file in my repository.  Then I can make a commit.

Then there's a restore shell script to copy things back to my system,
and restore ownership and permissions with setfacl.

I store the backup and restore scripts in the repository.  Paths are
currently hard-coded.  I'm sure there's a more flexible way to do
this, though I'd need some means of representing the correspondence
between content in the repository and files in my filesystem.


On 12/13/06, Daniel Barkalow [off-list ref] wrote:

On Tue, 12 Dec 2006, Kyle Moffett wrote:

quoted

Hmm, ok.  It would seem to be a reasonable requirement that if you want to
change any of the "preserve_*_attributes" config options you need to blow

away

quoted

and recreate your index, no?  I would probably change the underlying index
format pretty completely and stick a new version tag inside it.

You should be able to promote an insufficient-version index to a
new-version index that's needs to be refreshed for every entry. (And then
update-index would take care of the necessary rewrite-everything in the
normal way). But I suspect that the right thing is to require that the
repository be created with a "commits-include-directories-not-trees" flag,
and this means that you always use the extra-detailed index, and the
options only affect what information is filtered out in transit between
the directory object and the index. Having more information in the index
is merely a potential waste of space, not a correctness issue (we have
extra information for trees in the index now, remember); it just means
that there are more things that will cause git to reread the file, rather
than declaring it unchanged with a stat().

For that matter, it may be best for the directory objects to record what
information in them is real, and keep the "what's content" mask in the
index as well. If it changes over the history of a repository, you want to
correctly interpret the historical commits.

quoted

Ok, seems straightforward enough.  One other thing that crossed my mind

was

quoted

figuring out how to handle hardlinks.  The simplest solution would be to

add

quoted

an extra layer of indirection between the "file inode" and the "file

data".

quoted

Instead of your directory pointing to a "file-data" blob and

"file-attributes"

quoted

object, it would point to an "file-inode" object with embedded attribute

data

quoted

and a pointer to the file contents blob.

I remember reading some discussions from the early days of GIT about how

that

quoted

was considered and discarded because the extra overhead wouldn't give any

real

quoted

tangible benefit.  On the other hand for something like /etc the added
benefits of tracking extended attributes and hardlinks might outweigh the

cost

quoted

of a bunch of extra objects in the database.  A bit of care with the
construction of the index file should make it sufficiently efficient for
day-to-day usage.

I was thinking this could be internal to the directory object, but you
probably want to support hardlinks shared between dentries in different
directory objects, so you're probably right that this makes sense.

Alternatively, you could use a single "directory" object for the whole
state (including subdirectories), making hardlinks out of the object
clearly impossible, or you could use some scheme for sharing
sub-"directory" objects that would imply that hardlinks are within an
object (the hard part here is finding things when their locations aren't
predictable by name).

	-Daniel
*This .sig left intentionally blank*
-
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
epistemological humility

On Sun, 10 Dec 2006, Kyle Moffett wrote:

I've recently become somewhat interested in the idea of using GIT to store the
contents of various folders in /etc.  However after a bit of playing with
this, I discovered that GIT doesn't actually preserve all permission bits
since that would cause problems with the more traditional software development
model.  I'm curious if anyone has done this before; and if so, how they went
about handling the permissions and ownership issues.

I spent a little time looking over how GIT stores and compares permission
bits; trying to figure out if it's possible to patch in a new configuration
variable or two; say "preserve_all_perms" and "preserve_owner", or maybe even
"save_acls".  It looks like standard permission preservation is fairly basic;
you would just need to patch a few routines which alter the permissions read
in from disk or compare them with ones from the database.  On the other hand,
it would appear that preserving ownership or full POSIX ACLs might be a bit of
a challenge.

The first thing you'd want to do is correct the fact that the index 
doesn't keep full permissions. We decided long ago that we don't want to 
track more than 0100, but we're discarding the rest between the filesystem 
and the index, rather than between the index and the tree. (This is weird 
of us, since we keep gid and uid in the index, as changedness heuristics, 
but don't keep permissions; of course, we'd have to apply umask to the 
index when we check it out to sync what we expect to be there with what 
has actually been created.)

I think that would be the only change needed to the index and 
index/working directory connection, although it might be necessary to 
support longer values for uid/gid/etc, since they'd be important data now.

Note that git only stores content, not incidental information. But a lot 
of information which is incidental in a source tree is content in /etc. 
This implies that /etc and working/linux-2.6 are fundamentally different 
sorts of things, because different aspects of them are content.

I'd suggest a new object type for a directory with permissions, ACLs, and 
so forth. It should probably use symbolic owner and group, too. My guess 
is that you'll want to use "commit"s, the new object type, and "blob"s. 
Everything that uses trees would need to have a version that uses the new 
type. But I think that you generally want different behavior anyway, so 
that's not a major issue.

	-Daniel

On 12/10/06, Kyle Moffett [off-list ref] wrote:

quoted

I've recently become somewhat interested in the idea of using GIT  
to store the contents of various folders in /etc.  However after a  
bit of playing with this, I discovered that GIT doesn't actually  
preserve all permission bits since that would cause problems with  
the more traditional software development model.  I'm curious if  
anyone has done this before; and if so, how they went about  
handling the permissions and ownership issues.

I spent a little time looking over how GIT stores and compares  
permission bits; trying to figure out if it's possible to patch in  
a new configuration variable or two; say "preserve_all_perms" and  
"preserve_owner", or maybe even "save_acls".  It looks like  
standard permission preservation is fairly basic; you would just  
need to patch a few routines which alter the permissions read in  
from disk or compare them with ones from the database.  On the  
other hand, it would appear that preserving ownership or full  
POSIX ACLs might be a bit of a challenge.

On Dec 10, 2006, at 10:06:14, Santi Béjar wrote:

I have not used it, but you could try:

http://www.isisetup.ch/

that uses git as a backend.

Wow, umm, that's actually really interesting for me, given that I'm  
most interested in these sorts of things on Debian.  I can't find  
much documentation on their site; the tools look vaguely immature but  
I haven't really had much time to look at it yet.

On Dec 10, 2006, at 09:49:50, Jeff Garzik wrote:

It's a great idea, something I would like to do, and something I've  
suggested before.  You could dig through the mailing list archives,  
if you're motivated.

I have been digging through the archives; I was just holding out hope  
that somebody else on the list had already halfway beat me to the  
punch.  Guess not :-D

I actively use git to version, store and distribute an exim mail  
configuration across six servers.  So far my solution has been a  
'fix perms' script, or using the file perm checking capabilities of  
cfengine.

But it would be a lot better if git natively cared about ownership  
and permissions (presumably via an option).

I was thinking about a standard config option in the GIT config file,  
that way users could have a personal default and repositories could  
specify it locally.

I started tinkering but quickly discovered that permissions handling  
in general in GIT seems to be a mess; there's about 4 different tiers  
where permissions data is manipulated in various formats.  Some  
places use network-endian 16-bit values, there's a couple functions  
which do different truncations to 644 or 755 format.  There are 2  
functions which canonicalize the file mode based on symlink or  
directory status, each in subtly different ways.

I'm slowly sorting through things but if I could get a few pointers  
from someone intimately familiar with the code that would be most  
appreciated:  I'd like to try to add new entries to tree objects  
which older versions of GIT would ignore but which newer versions of  
GIT would use to store ACL or extended-attribute data.

The simplest solution which admittedly breaks the ability of older  
GITs to read the data from a file with attributes (ignoring the ext- 
attrs themselves) is to create a new "file-with-extended-attributes"  
object which contains a binary concatenation (with length bytes and  
attribute names and such) of the file and its extended attributes.   
That breaks the old GIT assumption that permission and security data  
is part of the directory not the file, but it's more in-line with the  
way extended attributes are attached to the inodes in the filesystem  
(although that doesn't really matter IMO).

Alternatively I might be able to add a new entry to each tree object  
with invalid extended file mods bits (IE: Neither a directory, a  
file, nor a symlink), or perhaps an entry with an empty name, which  
points to a new "extended attribute table".  That table could either  
map from (entry, attribute) => (data) or from (entry) =>  
((attribute,data),(attribute,data),[...]), depending on which would  
be more efficient.  It's essential that the overhead for non-ext-attr  
repositories is O(1) and ideally the overhead for a bunch of files  
with the same ext-attr is O(size-of-ext-attr) + O(number-of-files- 
with-that-attr), although that may vary depending on implementation.

Advice, opinions, problems, and "this-has-no-chance-of-ever-even- 
remotely-working" are all useful and welcome!

Cheers,
Kyle Moffett

Kyle Moffett wrote:

I've recently become somewhat interested in the idea of using GIT to 
store the contents of various folders in /etc.  However after a bit of 
playing with this, I discovered that GIT doesn't actually preserve all 
permission bits since that would cause problems with the more 
traditional software development model.  I'm curious if anyone has done 
this before; and if so, how they went about handling the permissions and 
ownership issues.

I spent a little time looking over how GIT stores and compares 
permission bits; trying to figure out if it's possible to patch in a new 
configuration variable or two; say "preserve_all_perms" and 
"preserve_owner", or maybe even "save_acls".  It looks like standard 
permission preservation is fairly basic; you would just need to patch a 
few routines which alter the permissions read in from disk or compare 
them with ones from the database.  On the other hand, it would appear 
that preserving ownership or full POSIX ACLs might be a bit of a challenge.

It's a great idea, something I would like to do, and something I've 
suggested before.  You could dig through the mailing list archives, if 
you're motivated.

I actively use git to version, store and distribute an exim mail 
configuration across six servers.  So far my solution has been a 'fix 
perms' script, or using the file perm checking capabilities of cfengine.

But it would be a lot better if git natively cared about ownership and 
permissions (presumably via an option).

	Jeff

Kyle Moffett wrote:

The simplest solution which admittedly breaks the ability of older  
GITs to read the data from a file with attributes (ignoring the ext- 
attrs themselves) is to create a new "file-with-extended-attributes"  
object which contains a binary concatenation (with length bytes and  
attribute names and such) of the file and its extended attributes.   
That breaks the old GIT assumption that permission and security data  
is part of the directory not the file, but it's more in-line with the  
way extended attributes are attached to the inodes in the filesystem  
(although that doesn't really matter IMO).

This contradict git philosophy of "tracking contents".

Alternatively I might be able to add a new entry to each tree object  
with invalid extended file mods bits (IE: Neither a directory, a  
file, nor a symlink), or perhaps an entry with an empty name, which  
points to a new "extended attribute table".  That table could either  
map from (entry, attribute) => (data) or from (entry) =>  
((attribute,data),(attribute,data),[...]), depending on which would  
be more efficient.  It's essential that the overhead for non-ext-attr  
repositories is O(1) and ideally the overhead for a bunch of files  
with the same ext-attr is O(size-of-ext-attr) + O(number-of-files- 
with-that-attr), although that may vary depending on implementation.

Wouldn't it be better to add another field in the tree object, that
instead of storing "(filemode, link to contents, name)" it would
store "(filemode, link to extended attributes, link to contents, name)"
where "filemode" is mode of a file of which git uses only a few bits
(is a directory, is a symlink, is a file, is a executable file),
and "link to" is sha1 of appropriate blob (or tree) object? Extended
attributes could be stored in new type of object, or just in blob
object. Well, you'd have to extend index in similar way (and add
a way to store extended attributes for directories in index; nowit only
stores info about files).

This of course breaks backwards compatibility...

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

On 12/13/06, Martin Langhoff [off-list ref] wrote:

I've been thinking myself of writing a pdumpfs lookalike that uses git
internally. Sounds you you've got one already ;-)

Actually - what I was considering was mixing the "daily commit" with
GITFS ;-) http://www.sfgoth.com/~mitch/linux/gitfs/

are your scripts published anywhere?

cheers,