[PATCH] doc: technical details about the index file format

This bases on the original work by Robin Rosenberg:

http://thread.gmane.org/gmane.comp.version-control.git/73471

Signed-off-by: Nguyễn Thái Ngọc Duy <redacted>
---
 I split index entry out so the overall format is clearer.

 Other changes:
 - mention of version 3
 - added ino and mode
 - added extended flags (v3)
 - entry sort order

 Again I don't realy know REUC extension, so only placeholder

 Documentation/technical/index-format.txt |  139 ++++++++++++++++++++++++++++++
 1 files changed, 139 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/technical/index-format.txt

diff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt
new file mode 100644
index 0000000..3e113ca
--- /dev/null
+++ b/Documentation/technical/index-format.txt

@@ -0,0 +1,139 @@
+GIT index format
+================
+
+= The git index file has the following format
+
+  All binary numbers are in network byte order. Version 2 is described
+  here unless stated otherwise.
+
+   - A 12-byte header consisting of
+
+     4-byte signature:
+       The signature is { 'D', 'I', 'R', 'C' }
+
+     4-byte version number:
+       The current supported versions are 2 and 3.
+
+     32-bit number of index entries.
+
+   - A number of sorted index entries
+
+   - Extensions
+
+     Extensions are identified by signature. Optional extensions can
+     be ignored if GIT does not understand them.
+
+     GIT currently supports tree cache and resolve undo extensions.
+
+     4-byte extension signature. If the first byte is 'A'..'Z' the
+     extension is optional and can be ignored.
+
+     32-bit size of the extension
+
+     Extension data
+
+   - 160-bit SHA-1 over the content of the index file before this
+     checksum.
+
+== Index entry
+
+  Index entries are sorted with memcmp() by entry name. Entries with
+  the same name are sorted by their stage.
+
+  32-bit ctime seconds, the last time a file's metadata changed
+    this is stat(2) data
+
+  32-bit ctime nanoseconds (modulo 1G)
+    this is stat(2) data
+
+  32-bit mtime seconds, the last time a file's data changed
+    this is stat(2) data
+
+  32-bit mtime nanoseconds (modulo 1G)
+    this is stat(2) data
+
+  32-bit dev
+    this is stat(2) data
+
+  32-bit ino
+    this is stat(2) data
+
+  32-bit mode, split into (high to low bits)
+
+    4-bit object type
+      valid values in binary are 1000 (blob), 1010 (symbolic link)
+      and 1110 (gitlink)
+
+    3-bit unused
+
+    9-bit unix permission (only 0755 and 0644 are valid)
+
+  32-bit uid
+    this is stat(2) data
+
+  32-bit gid
+    this is stat(2) data
+
+  32-bit file size
+    This is the on-disk size from stat(2)
+
+  160-bit SHA-1 for the represented object
+
+  A 16-bit field split into (high to low bits)
+
+    1-bit assume-valid flag
+
+    1-bit extended flag (must be zero in version 2)
+
+    2-bit stage (during merge)
+
+    12-bit name length if the length is less than 0x0FFF
+
+  (Version 3) A 16-bit field, only applicable if the "extended flag"
+  above is 1, split into (high to low bits).
+
+    1-bit reserved for future
+
+    1-bit skip-worktree flag (used by sparse checkout)
+
+    1-bit intent-to-add flag (used by "git add -N")
+
+    13-bit unused, must be zero
+
+  Entry path name (variable length) relative to top-level directory
+    (without leading slash). '/' is used as path separator. Special
+    paths ".", ".." and ".git" (without quotes) are disallowed.
+    Trailing slash is also disallowed.
+
+  1-8 nul bytes as necessary to pad the entry to a multiple ot eight bytes
+  while keeping the name NUL-terminated.
+
+== Extensions
+
+=== Tree cache
+
+  Tree cache extension contains pre-computes hashes for all trees that
+  can be derived from the index
+
+  - Extension tag { 'T', 'R', 'E', 'E' }
+
+  - 32-bit size
+
+  - A number of entries
+
+     NUL-terminated tree name
+
+     Blank-terminated ASCII decimal number of entries in this tree
+
+     Newline-terminated position of this tree in the parent tree. 0 for
+     the root tree
+
+     160-bit SHA-1 for this tree and it's children
+
+=== Resolve undo
+
+  TODO
+
+  - Extension tag { 'R', 'E', 'U', 'C' }
+
+  - 32-bit size

-- 
1.7.1.rc1.69.g24c2f7

Hi,

Nguyễn Thái Ngọc Duy writes:

This bases on the original work by Robin Rosenberg:

http://thread.gmane.org/gmane.comp.version-control.git/73471

[...]

It might be more profitable to mention the Message-ID instead.
[off-list ref]

-- Ram

Heya,

2010/9/1 Ramkumar Ramachandra [off-list ref]:

It might be more profitable to mention the Message-ID instead.
[off-list ref]

If you really want to do that, use this link:

http://mid.gmane.org/1202711335-12026-1-git-send-email-robin.rosenberg@dewire.com

-- 
Cheers,

Sverre Rabbelier

onsdagen den 1 september 2010 11.53.45 skrev  Nguyễn Thái Ngọc Duy:

This bases on the original work by Robin Rosenberg:

http://thread.gmane.org/gmane.comp.version-control.git/73471

No need for this. My name is enough

Signed-off-by: Nguyễn Thái Ngọc Duy <redacted>

Add
Signed-off-by: Nguyễn Thái Ngọc Duy <redacted>

Index entries are sorted in ascending order on the name field, interpreted as
a string of unsigned bytes.

+
+  32-bit ctime seconds, the last time a file's metadata changed
+    this is stat(2) data
+
+  32-bit ctime nanoseconds (modulo 1G)
+    this is stat(2) data
+
+  32-bit mtime seconds, the last time a file's data changed
+    this is stat(2) data
+
+  32-bit mtime nanoseconds (modulo 1G)
+    this is stat(2) data
+
+  32-bit dev
+    this is stat(2) data
+
+  32-bit ino
+    this is stat(2) data
+
+  32-bit mode, split into (high to low bits)
+
+    4-bit object type
+      valid values in binary are 1000 (blob), 1010 (symbolic link)
+      and 1110 (gitlink)
+
+    3-bit unused
+
+    9-bit unix permission (only 0755 and 0644 are valid)
+
+  32-bit uid
+    this is stat(2) data
+
+  32-bit gid
+    this is stat(2) data
+
+  32-bit file size
+    This is the on-disk size from stat(2)
+
+  160-bit SHA-1 for the represented object
+
+  A 16-bit field split into (high to low bits)
+
+    1-bit assume-valid flag
+
+    1-bit extended flag (must be zero in version 2)
+
+    2-bit stage (during merge)
+
+    12-bit name length if the length is less than 0x0FFF
+
+  (Version 3) A 16-bit field, only applicable if the "extended flag"
+  above is 1, split into (high to low bits).
+
+    1-bit reserved for future
+
+    1-bit skip-worktree flag (used by sparse checkout)
+
+    1-bit intent-to-add flag (used by "git add -N")
+
+    13-bit unused, must be zero
+
+  Entry path name (variable length) relative to top-level directory

...to the top level...

+    (without leading slash). '/' is used as path separator. Special

The special...

+    paths ".", ".." and ".git" (without quotes) are disallowed.
+    Trailing slash is also disallowed.

Why would anyone even consider adding a trailing slash to a _file_ name?

   The exact encoding is undefined, but the '.', and '/' characters
   are encoded in 7-bit ASCII and the encoding cannot contain a nul byte.
   Generally a superset of ASCII

+
+  1-8 nul bytes as necessary to pad the entry to a multiple ot eight bytes

...of eight bytes

A typo of mine.

+  while keeping the name NUL-terminated.
+
+== Extensions
+
+=== Tree cache
+
+  Tree cache extension contains pre-computes hashes for all trees that
+  can be derived from the index
+
+  - Extension tag { 'T', 'R', 'E', 'E' }
+
+  - 32-bit size
+
+  - A number of entries
+
+     NUL-terminated tree name
+
+     Blank-terminated ASCII decimal number of entries in this tree
+
+     Newline-terminated position of this tree in the parent tree. 0 for
+     the root tree
+
+     160-bit SHA-1 for this tree and it's children
+
+=== Resolve undo
+
+  TODO
+
+  - Extension tag { 'R', 'E', 'U', 'C' }
+
+  - 32-bit size

2010/9/2 Robin Rosenberg [off-list ref]:

quoted

+  Entry path name (variable length) relative to top-level directory

...to the top level...

quoted

+    (without leading slash). '/' is used as path separator. Special

The special...

quoted

+    paths ".", ".." and ".git" (without quotes) are disallowed.
+    Trailing slash is also disallowed.

Why would anyone even consider adding a trailing slash to a _file_ name?

Well, I was tempted to put directories in index more than once. And
subprojects are actually directories although they are treated as
files in index.
-- 
Duy

This bases on the original work by Robin Rosenberg.

Signed-off-by: Robin Rosenberg <redacted>
Signed-off-by: Nguyễn Thái Ngọc Duy <redacted>
---
 Fixups after Robin's review

 Documentation/technical/index-format.txt |  144 ++++++++++++++++++++++++++++++
 1 files changed, 144 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/technical/index-format.txt

diff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt
new file mode 100644
index 0000000..0285d88
--- /dev/null
+++ b/Documentation/technical/index-format.txt

@@ -0,0 +1,144 @@
+GIT index format
+================
+
+= The git index file has the following format
+
+  All binary numbers are in network byte order. Version 2 is described
+  here unless stated otherwise.
+
+   - A 12-byte header consisting of
+
+     4-byte signature:
+       The signature is { 'D', 'I', 'R', 'C' }
+
+     4-byte version number:
+       The current supported versions are 2 and 3.
+
+     32-bit number of index entries.
+
+   - A number of sorted index entries
+
+   - Extensions
+
+     Extensions are identified by signature. Optional extensions can
+     be ignored if GIT does not understand them.
+
+     GIT currently supports tree cache and resolve undo extensions.
+
+     4-byte extension signature. If the first byte is 'A'..'Z' the
+     extension is optional and can be ignored.
+
+     32-bit size of the extension
+
+     Extension data
+
+   - 160-bit SHA-1 over the content of the index file before this
+     checksum.
+
+== Index entry
+
+  Index entries are sorted in ascending order on the name field,
+  interpreted as a string of unsigned bytes. Entries with the same
+  name are sorted by their stage field.
+
+  32-bit ctime seconds, the last time a file's metadata changed
+    this is stat(2) data
+
+  32-bit ctime nanoseconds (modulo 1G)
+    this is stat(2) data
+
+  32-bit mtime seconds, the last time a file's data changed
+    this is stat(2) data
+
+  32-bit mtime nanoseconds (modulo 1G)
+    this is stat(2) data
+
+  32-bit dev
+    this is stat(2) data
+
+  32-bit ino
+    this is stat(2) data
+
+  32-bit mode, split into (high to low bits)
+
+    4-bit object type
+      valid values in binary are 1000 (blob), 1010 (symbolic link)
+      and 1110 (gitlink)
+
+    3-bit unused
+
+    9-bit unix permission (only 0755 and 0644 are valid)
+
+  32-bit uid
+    this is stat(2) data
+
+  32-bit gid
+    this is stat(2) data
+
+  32-bit file size
+    This is the on-disk size from stat(2)
+
+  160-bit SHA-1 for the represented object
+
+  A 16-bit field split into (high to low bits)
+
+    1-bit assume-valid flag
+
+    1-bit extended flag (must be zero in version 2)
+
+    2-bit stage (during merge)
+
+    12-bit name length if the length is less than 0x0FFF
+
+  (Version 3) A 16-bit field, only applicable if the "extended flag"
+  above is 1, split into (high to low bits).
+
+    1-bit reserved for future
+
+    1-bit skip-worktree flag (used by sparse checkout)
+
+    1-bit intent-to-add flag (used by "git add -N")
+
+    13-bit unused, must be zero
+
+  Entry path name (variable length) relative to top level directory
+    (without leading slash). '/' is used as path separator. The special
+    paths ".", ".." and ".git" (without quotes) are disallowed.
+    Trailing slash is also disallowed.
+
+    The exact encoding is undefined, but the '.' and '/' characters
+    are encoded in 7-bit ASCII and the encoding cannot contain a nul
+    byte. Generally a superset of ASCII.
+
+  1-8 nul bytes as necessary to pad the entry to a multiple of eight bytes
+  while keeping the name NUL-terminated.
+
+== Extensions
+
+=== Tree cache
+
+  Tree cache extension contains pre-computes hashes for all trees that
+  can be derived from the index
+
+  - Extension tag { 'T', 'R', 'E', 'E' }
+
+  - 32-bit size
+
+  - A number of entries
+
+     NUL-terminated tree name
+
+     Blank-terminated ASCII decimal number of entries in this tree
+
+     Newline-terminated position of this tree in the parent tree. 0 for
+     the root tree
+
+     160-bit SHA-1 for this tree and it's children
+
+=== Resolve undo
+
+  TODO
+
+  - Extension tag { 'R', 'E', 'U', 'C' }
+
+  - 32-bit size

-- 
1.7.1.rc1.69.g24c2f7

onsdagen den 1 september 2010 20.54.20 skrev  Robin Rosenberg:

onsdagen den 1 september 2010 11.53.45 skrev  Nguyễn Thái Ngọc Duy:

quoted

This bases on the original work by Robin Rosenberg:

http://thread.gmane.org/gmane.comp.version-control.git/73471

No need for this. My name is enough

quoted

Signed-off-by: Nguyễn Thái Ngọc Duy <redacted>

Add this rather than then one I sent in the previus mail...

Signed-off-by: Robin Rosenberg <redacted>

-- robin

2010/9/1 Nguyễn Thái Ngọc Duy [off-list ref]:

+== Index entry
+
+  Index entries are sorted in ascending order on the name field,
+  interpreted as a string of unsigned bytes. Entries with the same
+  name are sorted by their stage field.
+
+  32-bit ctime seconds, the last time a file's metadata changed
+    this is stat(2) data
+
+  32-bit ctime nanoseconds (modulo 1G)
+    this is stat(2) data

Maybe I'm missing something, but I failed to find where "modulo 1G" comes from.
AFAICS (read-cache.c), the stat data are saved almost unmodified
(casted to unsigned int).
(BTW, is 1G the Gravitational Constant or what?)

I'm not sure it is safe to assume that  every system Git will be
ported to defines
"unsigned int" to be 32 bits. OTOH, never met one where it is something else.
Still, using uint32_t (the POSIX types) in ondisk_cache_entry would be clearer
(unlikely alignment issues aside.

On Sep 2, 2010, at 1:56 AM, Alex Riesen wrote:

2010/9/1 Nguyễn Thái Ngọc Duy [off-list ref]:

quoted

+== Index entry
+
+  Index entries are sorted in ascending order on the name field,
+  interpreted as a string of unsigned bytes. Entries with the same
+  name are sorted by their stage field.
+
+  32-bit ctime seconds, the last time a file's metadata changed
+    this is stat(2) data
+
+  32-bit ctime nanoseconds (modulo 1G)
+    this is stat(2) data

Maybe I'm missing something, but I failed to find where "modulo 1G"  
comes from.
AFAICS (read-cache.c), the stat data are saved almost unmodified
(casted to unsigned int).
(BTW, is 1G the Gravitational Constant or what?)

G stands for "giga-" meaning one billion, so 1G refers to one billion  
nanoseconds.

I'm not sure it is safe to assume that  every system Git will be
ported to defines
"unsigned int" to be 32 bits. OTOH, never met one where it is  
something else.

DOS and early Mac compilers have used 16-bit ints, but I don't think  
anyone cares.

Josh