Host Configuration File

System configuration, especially when shared among a team and spread over multiple locations and machines, is cumbersome. A number of products try to address this problem space, usually referred to as System Configuration Management. Note of the solutions I was able to find were much to my liking, so here's my idea of how to do it properly

History

A couple of years back, I put together a system based on the FreeBSD build system: basically a set of Makefiles that will create all config files and install them in the right spots. These Makefiles and the template config files are then managed through CVS, giving the team the necessary tracebility to all configuration changes. It also provides some templating allowing certain config files to be automatically filled in with the necessary host-specific information. However, all changes need to be made to the source files and installed, so you can't easily fiddle with settings and then check the working config in. Or you would only realize a major mistake after lots of harm were done, because you made a typo in a Makefile and wiped out half the mail config. This system is still in use today, and is serving us well, but extending it to new services and machines is too cumbersome.

For another set of machines, I then decided to keep "complex" configs in CVS directly, by checking in the config directories. This mostly works, but for certain bits, the ownership and permissions don't work right, requiring some hacks or manual corrections. Also, since many of the files in the config space (/etc, /usr/local/etc) are system or package supplied, you don't want to put them on your own config tree unless you have to. So you get to manage .cvsignore a lot. Managing both cluster-wide and host-specific files reliably is cumbersome and invariably involves making copies by hand.

Design Goals and Principles

• Any new system should be as transparent to daily system administration as possible, and it should be really straightforward to understand what's happening when. When you're trying to track down a config problem, you don't want to worry about what the config system is doing to your files, or where bits live.
• No generation of files. While auto-generated or auto-updated files can be really useful, the complexity added is not worth it for our purposes. Most config files can be split up into multiple files (by means of includes, etc.), so having both host-specific and cluster-general config data at the same time is possible.
• Mixing of host-specific and hostgroup-specific config files must be possible. Some files can and should be identical on all systems of a cluster, and updating all machines should be easy. No fancy replication mechanism, just the ability to update a machine with a single command via ssh.
• SIngle-step operations should be the norm. When changing a configuration, commiting that change should be a single-step operation; it's hard enough to make the right change anyway, and saving that information should be easy.
• Only one copy of a file should exist. There should be no potential for confusion between the actual config file as used by the system, and any other copies the configuration management needs for it's internal processes. Ideally, there's only one copy: the master.
• One configuration for all admins. Individual admins have individual access to the machines and the repository, but they do not have private copies of files, and they share all configurations.

Hosts and Host Groups

Each managed system can be part of one or more host groups. Each host group defines a set of config files.

Each managed config file on a managed system is part of either the host-specific set of files, or part one of the files belonging to a host group. When adding a config file to the repository, you specify which file set it should be part of.

When updating from the repository, files are retrieved first from the host file set, then from the host group file sets. If a file is part of more than one file set, the most specific one has precedence (host file set, then the group file sets in the order listed in the host config).

Bundles and Repository Structure

In the file system, config files are scattered around. In the repository, they should be organized into a structure. A bundle is the top-level organizational unit. In the file system, it corresponds to a directory where config files are stored (e.g. /usr/local/etc/apache). In the repository, the bundle is a top-level directory.

The repository structure has at it's top level three directories: config, hosts and groups. The config directory stores meta-configuration information for the hosts that are using HCF. The hosts directory contains a directory for each managed host, and the groups directory contains one directory per host group. Each of these then contain the bundle directories, which in turn contain the actual config files and the associated metadata.

Commands

hcf update

Applies the lastest changes from the repository to the system files. Equivalent to cvs update.

You can specify one or more bundles or complete paths to update.

hcf commit

Commits the current state of the system files to the repository. Equivalent to cvs commit.

You can specify one or more bundles or complete paths to commit.