From: Junio C Hamano <hidden> Date: 2016-06-15 22:55:00
david@lang.hm writes:
I've got a directory tree that holds config data for all my
servers. This consists of one directory per server (which is updated
periodically from what is currently configured on that server), plus
higher level summary reports and similar information.
today I have just a single git tree covering everything, and I make a
commit each time one of the per-server directories is updated, and
again when the top-level stuff is created.
It is quite clear to me what you are keeping at the top-level files,
but if a large portion of the configuration for these servers are
shared, it might not be a bad idea to have a canonical "gold-master"
configuration branch, to which the shared updates are applied, with
a branch per server that forks from that canonical branch to keep
the machine specific tweaks as differences from the canonical stuff,
instead of having N subdirectories (one per machine).
I've got a directory tree that holds config data for all my
servers. This consists of one directory per server (which is updated
periodically from what is currently configured on that server), plus
higher level summary reports and similar information.
today I have just a single git tree covering everything, and I make a
commit each time one of the per-server directories is updated, and
again when the top-level stuff is created.
It is quite clear to me what you are keeping at the top-level files,
but if a large portion of the configuration for these servers are
shared, it might not be a bad idea to have a canonical "gold-master"
configuration branch, to which the shared updates are applied, with
a branch per server that forks from that canonical branch to keep
the machine specific tweaks as differences from the canonical stuff,
instead of having N subdirectories (one per machine).
In an ideal world yes, but right now these machines are updated by many
different tools (unforuntantly including 'vi'), so these directories
aren't the config to be pushed out to the boxes (i.e. what they should
be), it's instead an archived 'what is', the result of changes from all
the tools.
The systems are all built with a standard image, but the automation tools
I do have tend to push identical files out to many of the systems (or
files identical except for a couple of lines)
David Lang
On Sun, Oct 14, 2012 at 12:57 AM, [off-list ref] wrote:
On Sat, 13 Oct 2012, Junio C Hamano wrote:
quoted
david@lang.hm writes:
quoted
I've got a directory tree that holds config data for all my
servers. This consists of one directory per server (which is updated
periodically from what is currently configured on that server), plus
higher level summary reports and similar information.
today I have just a single git tree covering everything, and I make a
commit each time one of the per-server directories is updated, and
again when the top-level stuff is created.
It is quite clear to me what you are keeping at the top-level files,
but if a large portion of the configuration for these servers are
shared, it might not be a bad idea to have a canonical "gold-master"
configuration branch, to which the shared updates are applied, with
a branch per server that forks from that canonical branch to keep
the machine specific tweaks as differences from the canonical stuff,
instead of having N subdirectories (one per machine).
In an ideal world yes, but right now these machines are updated by many
different tools (unforuntantly including 'vi'), so these directories aren't
the config to be pushed out to the boxes (i.e. what they should be), it's
instead an archived 'what is', the result of changes from all the tools.
The systems are all built with a standard image, but the automation tools I
do have tend to push identical files out to many of the systems (or files
identical except for a couple of lines)
David,
Is there any particular reason you aren't using etckeeper?
--
-Drew Northup
--------------------------------------------------------------
"As opposed to vegetable or mineral error?"
-John Pescatore, SANS NewsBites Vol. 12 Num. 59
On Sun, Oct 14, 2012 at 12:57 AM, [off-list ref] wrote:
quoted
On Sat, 13 Oct 2012, Junio C Hamano wrote:
quoted
david@lang.hm writes:
quoted
I've got a directory tree that holds config data for all my
servers. This consists of one directory per server (which is updated
periodically from what is currently configured on that server), plus
higher level summary reports and similar information.
today I have just a single git tree covering everything, and I make a
commit each time one of the per-server directories is updated, and
again when the top-level stuff is created.
It is quite clear to me what you are keeping at the top-level files,
but if a large portion of the configuration for these servers are
shared, it might not be a bad idea to have a canonical "gold-master"
configuration branch, to which the shared updates are applied, with
a branch per server that forks from that canonical branch to keep
the machine specific tweaks as differences from the canonical stuff,
instead of having N subdirectories (one per machine).
In an ideal world yes, but right now these machines are updated by many
different tools (unforuntantly including 'vi'), so these directories aren't
the config to be pushed out to the boxes (i.e. what they should be), it's
instead an archived 'what is', the result of changes from all the tools.
The systems are all built with a standard image, but the automation tools I
do have tend to push identical files out to many of the systems (or files
identical except for a couple of lines)
David,
Is there any particular reason you aren't using etckeeper?
not really, I've thought of that as a tool for managing a single system.
Some of the data in configs is sensitive (and much of it is not in /etc),
but I guess I should be able to work around those issues.
I can e-mail 'patches' to the central server, but I'm then back to the
same question that I started out with.
How can I sanely organize all these different, but similar sets of files
on the central server?
David Lang
On Sat, Oct 20, 2012 at 10:34 PM, [off-list ref] wrote:
On Sat, 20 Oct 2012, Drew Northup wrote:
quoted
On Sun, Oct 14, 2012 at 12:57 AM, [off-list ref] wrote:
quoted
On Sat, 13 Oct 2012, Junio C Hamano wrote:
quoted
david@lang.hm writes:
quoted
today I have just a single git tree covering everything, and I make a
commit each time one of the per-server directories is updated, and
again when the top-level stuff is created.
if a large portion of the configuration for these servers are
shared, it might not be a bad idea to have a canonical "gold-master"
configuration branch, to which the shared updates are applied, with
a branch per server that forks from that canonical branch to keep
the machine specific tweaks
In an ideal world yes, but right now these machines are updated by many
different tools (unforuntantly including 'vi'), so
these directories aren't the config to be pushed out to the boxes, it's
instead an archived 'what is', the result of changes from all the tools.
So you need to save what is there before pulling changes from the
master. That's no different from doing development work on an active
code base.
quoted
David,
Is there any particular reason you aren't using etckeeper?
not really, I've thought of that as a tool for managing a single system.
Some of the data in configs is sensitive (and much of it is not in /etc),
but I guess I should be able to work around those issues.
.gitignore and symlinks have been employed at times to deal with that.
How can I sanely organize all these different, but similar sets of files on
the central server?
The reason I asked about etckeeper is that you could, with proper
security in place, push those up to branches in a shared repository
(set up using gitolite, for instance) and not loose information about
the files in the process. This would allow you to make your changes on
one system using vi or whatever else is convenient, push the change up
to the shared repo, cherry-pick it into the other branches (using a
full check-out of all of the branches someplace safe as a workspace),
and pull that change out to the other systems.
If you are just looking to gather configuration information in the
large and don't want to engage in any shared management schemes (which
may involve symlinks in seemingly odd places to /etc and such) you may
wish to look at the System Configuration Collector [1] [2] which is a
nicely organized tool designed specifically to gather just the
important (and not highly confidential) information about common
software on a server and present it (and changes to it) to the admin
in a sensible manner. It is outside of the "Git Universe" but it does
what it sounds like you are doing now (if not what you wish to be
doing).
(1) http://www.qnh.eu/scc/
(2) http://sourceforge.net/projects/sysconfcollect/
--
-Drew Northup
--------------------------------------------------------------
"As opposed to vegetable or mineral error?"
-John Pescatore, SANS NewsBites Vol. 12 Num. 59
On Sat, Oct 20, 2012 at 10:34 PM, [off-list ref] wrote:
quoted
On Sat, 20 Oct 2012, Drew Northup wrote:
quoted
On Sun, Oct 14, 2012 at 12:57 AM, [off-list ref] wrote:
quoted
On Sat, 13 Oct 2012, Junio C Hamano wrote:
quoted
david@lang.hm writes:
quoted
today I have just a single git tree covering everything, and I make a
commit each time one of the per-server directories is updated, and
again when the top-level stuff is created.
if a large portion of the configuration for these servers are
shared, it might not be a bad idea to have a canonical "gold-master"
configuration branch, to which the shared updates are applied, with
a branch per server that forks from that canonical branch to keep
the machine specific tweaks
In an ideal world yes, but right now these machines are updated by many
different tools (unforuntantly including 'vi'), so
these directories aren't the config to be pushed out to the boxes, it's
instead an archived 'what is', the result of changes from all the tools.
So you need to save what is there before pulling changes from the
master. That's no different from doing development work on an active
code base.
I think I've done a poor job of explaining my problem.
I'm not looking for tips on how to manage the systems themselves, I'm
looking for suggestions on how to manage this data that I'm already
gathering on this reporting server.
I have the problem that different departments have their own (different)
preferred tools for implementing changes. There are 6 different
departments that need to be involved with a single system to build and
maintain it. Each department has their 'standard' way of doing things. At
least two of these departments are using different, central configuration
(i.e. puppet like) tools.
As a result, I am not looking to pull changes from the central location.
I'm just trying to gather information and be able to produce reports about
the systems (Including "This is what all the different configs files on
this server were like at time X"). I'm not using the distributed features
of git at this time.
I've got existing tools that do a very similar job to what it sounds like
sysconfcollect does that gather the non-sensitive info from all my remote
machines and sends the data to my central server. These tools send an
update whenever 'significant' changes are made, and in addition do a
scheduled update to catch less significant changes.
On my central server I have the directory configs-current that then has a
subdirectory details/systemname for each system that contains all the
information about htat system (populated by scripts that parse apart the
data mentioned above)
In other files and directories in configs-current I have lots of more
global data and reports. This includes things like a report of every
interface on every machine, the IP address, does it have link, what speed
is it at, etc.
Right now I have one git tree for configs-current and each time I update a
details/systemname tree I do
git add -a configs-current/details/$systemname
git commit -m'system update from $servername'
then when I run the summary scripts I do
git add -a configs-current
git commit -m'summary update'
This has been working for a few years
However, trying to go back in history to find a change on one system is a
pain.
Right now the updates accumulate until I manually trigger a processing
cycle to update the files. I would like to make it so that the updates to
each system's details/systemname directory is done automatically as the
e-mail from that system arrives, and this could result in parallel
updates. I don't think that git will handle this well in one tree with the
existing process (different processes doing git add and git commits in
parallel will end up mixing their data)
As one big tree, this has lots of commits (a couple hundred each update),
and this is making it slow to try and track changes to a particular file
in a particular system.
I'm thinking that splitting the history tracking per-server should make
everything faster.
I'm wondering if I should do a subproject for each details/systemname
directory, or if there is something else I can do to make this tracking of
the data better.
Doing a single repository with lots of branches doesn't seem like it would
work as I need to get at the data from all the branches at the same time.
I guess I could do something with branches on one repository, with a
different worktree for each system, but that seems a bit fragile (one
command with the wrong environment variables and it coudl really tangle
things up)
David Lang