Re: metastore

David Härdeman [off-list ref] writes:

On Tue, Oct 02, 2007 at 08:53:01PM +0100, martin f krafft wrote:

quoted

also sprach David Härdeman [off-list ref] [2007.09.19.2016 +0100]:

quoted

But I agree, if any changes were made to git, I'd advocate adding
arbitrary attributes to files (much like xattrs) in name=value
pairs, then any extended metadata could be stored in those
attributes and external scripts/tools could use them in some way
that makes sense...and also make sure to only update them when it
makes sense.

So where would those metdata be stored in your opinion?

I'm not sufficiently versed in the internals of git to have an
informed opinion :)

I think we have something like a length count for file names in index
and/or tree.  We could just put the (sorted) attributes after a NUL
byte in the file name and include them in the count.  It would also
make those artificially longer file names work more or less when
sorting them for deltification.

However, this requires implementing _policies_: it must be possible to
specify per repository exactly what will and what won't get tracked,
or one will get conflicts that are not necessary or appropriate.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

On Tue, 2 Oct 2007, David Kastrup wrote:

David Härdeman [off-list ref] writes:

quoted

On Tue, Oct 02, 2007 at 08:53:01PM +0100, martin f krafft wrote:

quoted

also sprach David Härdeman [off-list ref] [2007.09.19.2016 +0100]:

quoted

But I agree, if any changes were made to git, I'd advocate adding
arbitrary attributes to files (much like xattrs) in name=value
pairs, then any extended metadata could be stored in those
attributes and external scripts/tools could use them in some way
that makes sense...and also make sure to only update them when it
makes sense.

So where would those metdata be stored in your opinion?

I'm not sufficiently versed in the internals of git to have an
informed opinion :)

I think we have something like a length count for file names in index
and/or tree.  We could just put the (sorted) attributes after a NUL
byte in the file name and include them in the count.  It would also
make those artificially longer file names work more or less when
sorting them for deltification.

the problem with this is dealing with the attributes outside of git 
(especially when the filesystem can't store the attributes nativly, 
specificly including things like owners when not running as root)

this is one of the reasons for talking about useing a seperate file for 
the attributes (the other being the ability to minimize the impact to 
git-core of tracking attributes)

David Lang

also sprach david@lang.hm [off-list ref] [2007.10.02.2118 +0100]:

the problem with this is dealing with the attributes outside of git 
(especially when the filesystem can't store the attributes nativly, 
specificly including things like owners when not running as root)

In which case you should not be able to manipulate them (as you
could not test the result) and any commits could not affect them,
meaning they'd just stay unchanged.

-- 
martin;              (greetings from the heart of the sun.)
  \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck
 
the unix philosophy basically involves
giving you enough rope to hang yourself.
and then some more, just to be sure.
 
spamtraps: madduck.bogus@madduck.net

On Tue, 2 Oct 2007, martin f krafft wrote:

also sprach david@lang.hm [off-list ref] [2007.10.02.2118 +0100]:

quoted

the problem with this is dealing with the attributes outside of git
(especially when the filesystem can't store the attributes nativly,
specificly including things like owners when not running as root)

In which case you should not be able to manipulate them (as you
could not test the result) and any commits could not affect them,
meaning they'd just stay unchanged.

two problems with this

1. you do want to be able to manipulate them

1a. how do you reconcile a conflict during a merge?

2. git is a series of snapshots, what does it mean to 'stay unchanged'?

David Lang

also sprach david@lang.hm [off-list ref] [2007.10.02.2129 +0100]:

1. you do want to be able to manipulate them

1a. how do you reconcile a conflict during a merge?

How could there be a conflict if you can't make local changes
because you can't represent the attributes locally/natively?

2. git is a series of snapshots, what does it mean to 'stay unchanged'?

In simple terms, let (content,A,B) be an object with content
"content" and extended attributes A,B, and B cannot be represented
locally, but a new object is committed with a change to attribute
A (content2,A2), then the result is (content2,A2,B), as B simply
comes from the (corresponding object of the) parent.

Or am I totally misunderstanding?

-- 
martin;              (greetings from the heart of the sun.)
  \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck
 
when compared to windoze, unix is an operating system.
 
spamtraps: madduck.bogus@madduck.net

On Tue, 2 Oct 2007, martin f krafft wrote:

also sprach david@lang.hm [off-list ref] [2007.10.02.2129 +0100]:

quoted

1. you do want to be able to manipulate them

1a. how do you reconcile a conflict during a merge?

How could there be a conflict if you can't make local changes
because you can't represent the attributes locally/natively?

you merge two uptream branches that disagree about the attributes

quoted

2. git is a series of snapshots, what does it mean to 'stay unchanged'?

In simple terms, let (content,A,B) be an object with content
"content" and extended attributes A,B, and B cannot be represented
locally, but a new object is committed with a change to attribute
A (content2,A2), then the result is (content2,A2,B), as B simply
comes from the (corresponding object of the) parent.

Or am I totally misunderstanding?

it's very possible that I am misunderstanding, but do we really want to 
have to go back to the parent to duplicate things when creating a new 
commit?

and aren't you supposed to be able to have more then one parent? if you 
do, which one would you use?

David Lang

On Tue, Oct 02, 2007 at 10:04:56PM +0200, David Kastrup wrote:

David Härdeman [off-list ref] writes:

quoted

On Tue, Oct 02, 2007 at 08:53:01PM +0100, martin f krafft wrote:

quoted

also sprach David Härdeman [off-list ref] [2007.09.19.2016 +0100]:

quoted

But I agree, if any changes were made to git, I'd advocate adding
arbitrary attributes to files (much like xattrs) in name=value
pairs, then any extended metadata could be stored in those
attributes and external scripts/tools could use them in some way
that makes sense...and also make sure to only update them when it
makes sense.

So where would those metdata be stored in your opinion?

I'm not sufficiently versed in the internals of git to have an
informed opinion :)

I think we have something like a length count for file names in index
and/or tree.  We could just put the (sorted) attributes after a NUL
byte in the file name and include them in the count.  It would also
make those artificially longer file names work more or less when
sorting them for deltification.

Or perhaps the index format could be extended to include a new field for 
value=name pairs instead of overloading the name field.

But as I said, I have no idea how feasible it would be to change git to 
support another arbitrary length field in the index/tree file.

However, this requires implementing _policies_: it must be possible to
specify per repository exactly what will and what won't get tracked,
or one will get conflicts that are not necessary or appropriate.

I think the opposite approach would be better. Let git provide 
set/get/delete attribute operations and leave it at that. Then external 
programs can do what they want with that data and add/remove/modify tags 
as necessary (and also include the smarts to not, e.g. remove the 
permissions on all files if the git repo is checked out to a FAT fs).

-- 
David Härdeman

also sprach david@lang.hm [off-list ref] [2007.10.02.2154 +0100]:

quoted

How could there be a conflict if you can't make local changes
because you can't represent the attributes locally/natively?

you merge two uptream branches that disagree about the attributes

You win. :)

-- 
martin;              (greetings from the heart of the sun.)
  \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck
 
"mein gott, selbst ein huhn kann debian installieren, wenn du genug
 koerner auf die enter-taste legst."
                       -- thomas koehler in de.alt.sysadmin.recovery
 
spamtraps: madduck.bogus@madduck.net

also sprach David Härdeman [off-list ref] [2007.10.02.2215 +0100]:

I think the opposite approach would be better. Let git provide
set/get/delete attribute operations and leave it at that.

I like that idea.

-- 
martin;              (greetings from the heart of the sun.)
  \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck
 
"information superhighway"
 is just an anagram for
"i'm on a huge wispy rhino fart".
 
spamtraps: madduck.bogus@madduck.net

On Tue, 2 Oct 2007, David Härdeman wrote:

On Tue, Oct 02, 2007 at 10:04:56PM +0200, David Kastrup wrote:

quoted

David Härdeman [off-list ref] writes:

quoted

 On Tue, Oct 02, 2007 at 08:53:01PM +0100, martin f krafft wrote:

quoted

also sprach David Härdeman [off-list ref] [2007.09.19.2016 +0100]:

quoted

 But I agree, if any changes were made to git, I'd advocate adding
 arbitrary attributes to files (much like xattrs) in name=value
 pairs, then any extended metadata could be stored in those
 attributes and external scripts/tools could use them in some way
 that makes sense...and also make sure to only update them when it
 makes sense.

So where would those metdata be stored in your opinion?

 I'm not sufficiently versed in the internals of git to have an
 informed opinion :)

I think we have something like a length count for file names in index
and/or tree.  We could just put the (sorted) attributes after a NUL
byte in the file name and include them in the count.  It would also
make those artificially longer file names work more or less when
sorting them for deltification.

Or perhaps the index format could be extended to include a new field for 
value=name pairs instead of overloading the name field.

But as I said, I have no idea how feasible it would be to change git to 
support another arbitrary length field in the index/tree file.

quoted

However, this requires implementing _policies_: it must be possible to
specify per repository exactly what will and what won't get tracked,
or one will get conflicts that are not necessary or appropriate.

I think the opposite approach would be better. Let git provide set/get/delete 
attribute operations and leave it at that. Then external programs can do what 
they want with that data and add/remove/modify tags as necessary (and also 
include the smarts to not, e.g. remove the permissions on all files if the 
git repo is checked out to a FAT fs).

You need more than that.  You need to be able to log, blame etc on the 
attributes.  One of the big annoyances of Subversion properties is being 
unable to find out when or why a property value was changed.

I still don't see why the attributes need to be stored in git directly - 
particularly if you are going to use an external program to actually apply 
any settings - why not store the attributes as normal file (or files) of 
some sort tracked by git?  You could use any number of methods - e.g. use 
an sqlite database stored in the root of your tree, or a .<name>.props 
file alongside each path that you have properties for.  You could even 
write a system that uses such a method and was then SCM agnostic, allowing 
you to keep your attribute tracking system if/when something better than 
git comes along - or simply share it with less-fortunate souls stuck in an 
inferior system.

-- 
Julian

  ---
A strong conviction that something must be done is the parent of many
bad measures.
 		-- Daniel Webster

On Wed, 3 Oct 2007, Julian Phillips wrote:

Subject: Re: metastore

On Tue, 2 Oct 2007, David Härdeman wrote:

quoted

On Tue, Oct 02, 2007 at 10:04:56PM +0200, David Kastrup wrote:

quoted

David Härdeman [off-list ref] writes:

quoted

 On Tue, Oct 02, 2007 at 08:53:01PM +0100, martin f krafft wrote:

quoted

also sprach David Härdeman [off-list ref] [2007.09.19.2016 

+0100]:

quoted

quoted

quoted

 But I agree, if any changes were made to git, I'd advocate adding
 arbitrary attributes to files (much like xattrs) in name=value
 pairs, then any extended metadata could be stored in those
 attributes and external scripts/tools could use them in some way
 that makes sense...and also make sure to only update them when it
 makes sense.

quoted

So where would those metdata be stored in your opinion?

 I'm not sufficiently versed in the internals of git to have an

 informed opinion :)

I think we have something like a length count for file names in index
and/or tree.  We could just put the (sorted) attributes after a NUL
byte in the file name and include them in the count.  It would also
make those artificially longer file names work more or less when
sorting them for deltification.

Or perhaps the index format could be extended to include a new field for 
value=name pairs instead of overloading the name field.

But as I said, I have no idea how feasible it would be to change git to 
support another arbitrary length field in the index/tree file.

quoted

However, this requires implementing _policies_: it must be possible to
specify per repository exactly what will and what won't get tracked,
or one will get conflicts that are not necessary or appropriate.

I think the opposite approach would be better. Let git provide 
set/get/delete attribute operations and leave it at that. Then external 
programs can do what they want with that data and add/remove/modify tags as 
necessary (and also include the smarts to not, e.g. remove the permissions 
on all files if the git repo is checked out to a FAT fs).

You need more than that.  You need to be able to log, blame etc on the 
attributes.  One of the big annoyances of Subversion properties is being 
unable to find out when or why a property value was changed.

I still don't see why the attributes need to be stored in git directly - 
particularly if you are going to use an external program to actually apply 
any settings - why not store the attributes as normal file (or files) of some 
sort tracked by git?  You could use any number of methods - e.g. use an 
sqlite database stored in the root of your tree, or a .<name>.props file 
alongside each path that you have properties for.  You could even write a 
system that uses such a method and was then SCM agnostic, allowing you to 
keep your attribute tracking system if/when something better than git comes 
along - or simply share it with less-fortunate souls stuck in an inferior 
system.

one other big advantage of keeping things in a normal file, it's easier to 
get the results accepted into git!

don't forget that the core git maintainers don't really see this as a 
worthwhile effort, so the more intrusive the result is the less likely it 
is to be accepted. It may end up that storing the attributes inside of git 
_is_ the best thing to do, but it's gong to be a whole lot easier to get a 
patch to implement this accepted if it's a migration from an existing, 
heavily used, implementation then if it's from the 'outside' with people 
saying "this is a neat thing, we think people would use it if it only had 
this"

and even if an internal implementation does end up being the right thing, 
the exact shape of the API is an item that will require a lot of debate 
(and probably a few false starts) to get right. let's figure out the 
real-world useage patterns first, and then work from there as appropriate.

shifting back onto implementaion details

in the discussion a few weeks ago I was told that there is a way to look 
at the contents of a file that hasn't been checked out yet (somehow it 
exists in a useable form 'in the index') but when I asked for information 
about how to do this I never got a response.

the reason for needing this is that the routines writing the files need to 
be able to access this information when they are dong so, but that file 
may not be checked out.

for that matter, .gitattributes should have a similar problem (if 
.gitattibutes for a directory hasn't been checked out yet how do you know 
if you could do the line ending conversions on a file or not?). how is the 
problem addressed there? (or is it the case that all the use so far has 
really not used the per-directory files and everything is in the master 
file, and that doesn't change enough to find these problems?

David Lang

Hi,

On Tue, 2 Oct 2007, david@lang.hm wrote:

in the discussion a few weeks ago I was told that there is a way to look 
at the contents of a file that hasn't been checked out yet (somehow it 
exists in a useable form 'in the index') but when I asked for 
information about how to do this I never got a response.

git show :<filename>

(Note the ":")

Hth,
Dscho