Git files data formats documentation

This information may be useful for reading and writing the various Git 
files.

You might make some notes about old format headers and new format ones
and the use_legacy_headers flag.

I started off looking at packs so I knew about TYPE_AND_BASE128_SIZE.
Next I wanted to write objects so I looked at sha1_file.c. If you
don't look at the code closely write_binary_header() will lead you to
believe that object files use TYPE_AND_BASE128_SIZE. It took me a
couple of hours to notice use_legacy_headers and discover that it
defaults to on.

Jon Smirl
jonsmirl@gmail.com

A Large Angry SCM [off-list ref] wrote:

This information may be useful for reading and writing the various Git 
files.

[snip]

	#	/* -  -  -  -  -  -  -  -  -  -  -  - *\
	#	 | This reflects version 3 pack files |
	#	\* -  -  -  -  -  -  -  -  -  -  -  - */

[snip]

	#	/* -  -  -  -  -  -  -  -  -  -  -  - *\
	#	 | This reflects version 2 pack files |
	#	\* -  -  -  -  -  -  -  -  -  -  -  - */

Thanks for taking the time to write these out.  The pack delta
formats were particularly helpful as it caused me to go back
and look at the unpacking code in jgit.

Apparently I wasn't handling the version 2 pack file correctly as I
didn't support copy-from-result; I had an infinite loop if the base
didn't decompress in one read (never happen right now, but could
in the future); and apparently my insert opcode implementation was
causing an infinite loop.  Nasty bugs.  I need to get more unit
tests written apparently.  :-)


-- 
Shawn.

Shawn Pearce wrote:

A Large Angry SCM [off-list ref] wrote:

quoted

This information may be useful for reading and writing the various Git 
files.

[snip]

quoted

	#	/* -  -  -  -  -  -  -  -  -  -  -  - *\
	#	 | This reflects version 3 pack files |
	#	\* -  -  -  -  -  -  -  -  -  -  -  - */

[snip]

quoted

	#	/* -  -  -  -  -  -  -  -  -  -  -  - *\
	#	 | This reflects version 2 pack files |
	#	\* -  -  -  -  -  -  -  -  -  -  -  - */

Thanks for taking the time to write these out.  The pack delta
formats were particularly helpful as it caused me to go back
and look at the unpacking code in jgit.

Apparently I wasn't handling the version 2 pack file correctly as I
didn't support copy-from-result; I had an infinite loop if the base
didn't decompress in one read (never happen right now, but could
in the future); and apparently my insert opcode implementation was
causing an infinite loop.  Nasty bugs.  I need to get more unit
tests written apparently.  :-)

Keep in mind that the git-core code for reading version 2 or version 3 
pack files does _not_ handle copy-from-result correctly.

A Large Angry SCM wrote:

<TREE_ENTRY>
        # The type of the object referenced MUST be appropriate for
        # the mode. Regular files and symbolic links reference a BLOB
        # and directories reference a TREE.
        :       <OCTAL_MODE> <SP> <NAME> <NUL> <BINARY_OBJ_ID>
        ;

[...]

<OCTAL_MODE>
        # Octal encoding, without prefix, of the file system object
        # type and permission bits. The bit layout is according to the
        # POSIX standard, with only regular files, directories, and
        # symbolic links permitted. The actual permission bits are
        # all zero except for regular files. The only permission bit
        # of any consequence to Git is the owner executable bit. By
        # default, the permission bits for files will be either 0644
        # or 0755, depending on the owner executable bit.
        ;

I do wonder why there is <OCTAL_MODE> (and not <BINARY_OCTAL_MODE>) 
but <BINARY_OBJ_ID> (and not <HEX_OBJ_ID>).

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Jakub Narebski wrote:

A Large Angry SCM wrote:

quoted

<TREE_ENTRY>
        # The type of the object referenced MUST be appropriate for
        # the mode. Regular files and symbolic links reference a BLOB
        # and directories reference a TREE.
        :       <OCTAL_MODE> <SP> <NAME> <NUL> <BINARY_OBJ_ID>
        ;

[...]

quoted

<OCTAL_MODE>
        # Octal encoding, without prefix, of the file system object
        # type and permission bits. The bit layout is according to the
        # POSIX standard, with only regular files, directories, and
        # symbolic links permitted. The actual permission bits are
        # all zero except for regular files. The only permission bit
        # of any consequence to Git is the owner executable bit. By
        # default, the permission bits for files will be either 0644
        # or 0755, depending on the owner executable bit.
        ;

I do wonder why there is <OCTAL_MODE> (and not <BINARY_OCTAL_MODE>) 
but <BINARY_OBJ_ID> (and not <HEX_OBJ_ID>).

<OCTAL_MODE> because it's an ASCII string. <BINARY_OBJ_ID> because it's 
the 20 byte digest.

A Large Angry SCM wrote:

This information may be useful for reading and writing the various Git 
files.

Revised.

A Large Angry SCM wrote:

Jakub Narebski wrote:

quoted

I do wonder why there is <OCTAL_MODE> (and not <BINARY_OCTAL_MODE>) 
but <BINARY_OBJ_ID> (and not <HEX_OBJ_ID>).

<OCTAL_MODE> because it's an ASCII string. <BINARY_OBJ_ID> because it's 
the 20 byte digest.

I meant why git use ASCII string for octal mode, while using 20 byte digest
for object-id in tree format. More consistent would be use binary and
binary, or ASCII and ASCII (i.e. <HEX_OBJ_ID>).

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

A Large Angry SCM wrote:

This information may be useful for reading and writing the various Git
files.

[...]

        # For version 2 pack files, the size of a copy is limited to
        # 64K bytes or less and bit 6 of the opcode byte is set if the
        # source of the copy is from the buffer of the result object
        # instead of the the base object.
        #
        # It's unknown if any version 2 pack files were created with
        # bit 6 set in the opcode byte; however, the change that added
        # support for version 3 pack files removed the code that would
        # change the copy source to the result buffer.

There were no version 2 pack files with bit 6 set in the opcode byte 
ever produced (except on my own hard disk when I was experimenting with 
that feature).  The (negative) compression gain turned up to be not 
worth the needed computational cost to make use of it, hence that bit is 
now dedicated to specifying an extra size byte.

See commit d60fc1c8649f80c006b9f493c542461e81608d4b log message for 
more.


Nicolas