On the brink

Using puppet on Gentoo

2008-05-18T07:28:00.006+02:00

Puppet is a tool for managing your system configuration. It provides a complete language for expressing and realizing system settings. After some introductory words this post will focus on a Gentoo specific puppet module for managing package installations.

If you have no clue about puppet you might wish to read the introduction if you are interested in managing the configurations of your system in an efficient way. The discussion about the gentoo specific module will only be of interest to you if you already know the basics of writing puppet modules.

Introduction

What are the advantages of using puppet rather than editing all files in /etc by hand?

Using puppet means you create a repository of your configuration knowledge
You can replicate all of or part of the settings to another host
In addition you can version control and share your knowledge in a repository

Mind you: If you are only managing a single host you might not find much value in the items listed above. Indeed puppet only becomes useful if you really wish to apply a complex configuration over many hosts.

But of course this is true for any groupware server and in particular the Kolab Server. Porting Kolab to Gentoo is a project I have been working on for more than three years now.

The initial version (Kolab2/Gentoo-2.1) failed to make me really happy. One central reason for that has been the configuration tool provided by Kolab. While it works fine for the original version of the Kolab Server it simply fails to cope with the amount of options users have on Gentoo.

I always wanted to merge my own crappy tool for configuration management with the code from the Kolab Server. But a kind anonymous voice answered to the blog post linked in the previous setence, telling me that this is a stupid idea and I should use puppet. He was right.

So I'm establishing the Kolab2/Gentoo groupware server configuration based on puppet at the moment. As this includes generating some Gentoo specific modules for puppet it is now time to stop the introductory words and get down to some puppet code.

Installing packages for generic distributions

In order to tell puppet that you wish to have a single package installed you would use a construct like this:

package { openldap:
  ensure   => 'latest',
}

This works fine on most distributions but on Gentoo you might ask about support for use flags, keywords and masking.

Installing packages on Gentoo

My solution is the puppet module os_gentoo.

This module is mainly concerned with management of the files/directories you find at /etc/portage/package.* in your Gentoo system. In order for puppet to manage these paths it makes sense to convert these into directories.

The module provides four central parts:

Backup of the original contents of /etc/portage/package.* if these were files.
Converting the paths into directories.
Restoring the original file contents as /etc/portage/package.*/package.*.original.
Providing functions to easily manage use flags, keywords and masking for other packages.

Backup of /etc/portage/package.*

If the user managed /etc/portage/package.* as files we need to grab the content and store it. Puppet provides the file() function for that but that function will fail if it sees a directory. So we need to determine if the path already is a directory. We need to write some ruby code at this point and create a new fact:

# Determine if these are regular files
 
package_use = '/etc/portage/package.use'
 
Facter.add('use_isfile') do
  setcode do
    if FileTest.file?(package_use)
      true
    else
      false
    end
  end
end

...

Facts are little pieces of system information that puppet determines automatically using the tool dev-ruby/facter. The code given above checks if /etc/portage/package.use is a file and places that information in the variable use_isfile. We will shortly meet that variable again.

This fact is something we store as a plugin at os_gentoo/plugins/facter/portage_dirs.rb within the module.

The code actually performing the backup is packaged in a puppet class:

# Class gentoo::etc::portage::backup
#
# Stores user settings in the /etc/portage/package.* files.
#
# @author Gunnar Wrobel 
# @version 1.0
# @package os_gentoo
#
class gentoo::etc::portage::backup
{
  if $use_isfile {
    $use = file('/etc/portage/package.use')
  } else {
    $use = false
  }
  if $keywords_isfile {
    $keywords = file('/etc/portage/package.keywords')
  } else {
    $keywords = false
  }
  if $mask_isfile {
    $mask = file('/etc/portage/package.mask')
  } else {
    $mask = false
  }
  if $unmask_isfile {
    $unmask = file('/etc/portage/package.unmask')
  } else {
    $unmask = false
  }
}

Here we meet the variables again. In case $use_isfile is true the file contents will be parsed into $use. Otherwise the variable is set to false. We return to our backup two sections further down.

Converting /etc/portage/package.* into directories

Now that we have saved the file contents we can safely convert the files into directories. Puppet would not destroy the original files but instead store them in an archive. But recovering them from there would be cumbersome for the user. Automating the conversion seems to be a better solution.

Requiring a path to be a directory is easy in puppet:

# Class gentoo::etc::portage
#
# Ensure that all /etc/portage/package.* locations are actually
# handled as directories. This allows to easily manage the package
# specific settings for Gentoo.
#
# @author Gunnar Wrobel 
# @version 1.0
# @package os_gentoo
#
class gentoo::etc::portage
{
  # Check that we are able to handle /etc/portage/package.* as
  # directories
 
  file { 'package.use::directory':
    path => '/etc/portage/package.use',
    ensure => 'directory',
    tag => 'buildhost'
  }
 
  file { 'package.keywords::directory':
    path => '/etc/portage/package.keywords',
    ensure => 'directory',
    tag => 'buildhost'
  }
 
  file { 'package.mask::directory':
    path => '/etc/portage/package.mask',
    ensure => 'directory',
    tag => 'buildhost'
  }
 
  file { 'package.unmask::directory':
    path => '/etc/portage/package.unmask',
    ensure => 'directory',
    tag => 'buildhost'
  }
}

Again the four actions have been packaged into a single puppet class. The different actions all have a buildhost tag. This is only required if you really use a build host structure with your servers and plays no role otherwise.

Restoring the original /etc/portage/package.*

Now that puppet converted /etc/portage/package.* to directories we lost the original file contents. Another class will rescue them:

# Class gentoo::etc::portage::restore
#
# Restores user settings from the /etc/portage/package.* files.
#
# @author Gunnar Wrobel 
# @version 1.0
# @package os_gentoo
#
class gentoo::etc::portage::restore
{
  if $gentoo::etc::portage::backup::use {
    file { '/etc/portage/package.use/package.use.original':
      content => $gentoo::etc::portage::backup::use,
      tag => 'buildhost'
    }
  }
  if $gentoo::etc::portage::backup::keywords {
    file { '/etc/portage/package.keywords/package.keywords.original':
      content => $gentoo::etc::portage::backup::keywords,
      tag => 'buildhost'
    }
  }
  if $gentoo::etc::portage::backup::mask {
    file { '/etc/portage/package.mask/package.mask.original':
      content => $gentoo::etc::portage::backup::mask,
      tag => 'buildhost'
    }
  }
  if $gentoo::etc::portage::backup::unmask {
    file { '/etc/portage/package.unmask/package.unmask.original':
      content => $gentoo::etc::portage::backup::unmask,
      tag => 'buildhost'
    }
  }
}

For each of the four paths the original backup variable (e.g. $gentoo::etc::portage::backup::use) is checked for content. We need to use the full class path here to access the variable content. If it contains content it will be written to the corresponding new path (e.g. /etc/portage/package.use/package.use.original).

Handling /etc/portage/package.* with puppet

Now the management of /etc/portage/package.* becomes easy as puppet can place new files for every package or set of packages that requires special use flags, keywords or masking.

This is an example for the use flags:

# Function gentoo_use_flags
#
# Specify use flags for a package.
#
# @param context A unique context for the package
# @param package The package atom
# @param use The use flags to apply
#
define gentoo_use_flags ($context = '',
                         $package = '',
                         $use = '')
{
 
  file { "/etc/portage/package.use/${context}":
    content => "$package $use",
    require => File['package.use::directory'],
    tag => 'buildhost'
  }
 
}

The function takes a context which must be unique and will be used as path component. In addition the package atom needs to be specified including the use flags to be set. Puppet will then create a new file within /etc/portage/package.use using the file type (This is something different than the file function mentioned above).

The only new thing here is the require argument that specifies that puppet must ensure that the file operation with the name package.use::directory has been executed before creating this new file. In other words we ensure that /etc/portage/package.use is indeed a directory.

Managing package installations on Gentoo

Taking all these definitions together we can now express a package installation in the following way:

# Package installation
  case $operatingsystem {
    gentoo:
    {
      gentoo_unmask { openldap:
        context => 'service_openldap',
        package => '=net-nds/openldap-2.4.7',
        tag => 'buildhost'
      }
      gentoo_keywords { openldap:
        context => 'service_openldap',
        package => '=net-nds/openldap-2.4.7',
        keywords => "~$keyword",
        tag => 'buildhost'
      }
      gentoo_use_flags { openldap:
        context => 'service_openldap',
        package => 'net-nds/openldap',
        use => 'berkdb crypt overlays perl ssl syslog -sasl',
        tag => 'buildhost'
      }
      package { openldap:
        category => 'net-nds',
        ensure => 'latest',
        require => [ Gentoo_unmask['openldap'],
                       Gentoo_keywords['openldap'],
                       Gentoo_use_flags['openldap'] ],
        tag => 'buildhost'
      }
    }
    default:
    {
      package { openldap:
        ensure => 'installed',
      }
    }
  }
}

The example installs the experimental net-nds/openldap-2.4.7 package. We differentiate between Gentoo and other distributions using the $operatingsystem variable automatically provided by puppet.

Of course the Gentoo installation looks much more complex than the standard installation on other systems but we have a lot more flexibility on Gentoo. And the idea of the module is to allow us to use this flexibility within puppet.

The first three sections (gentoo_unmask,gentoo_keywords, and gentoo_use_flags) handle the settings in /etc/portage/package.* and the actual installation happens in the fourth section. We use the standard package type here but require that all the settings in /etc/portage/package.* have been executed before puppet runs emerge

A final note on the variable $keyword that is being used in the section above. This is another fact that prevents us from specifying keywords like ~x86 while we actually want ~amd64. It simply reads ACCEPT_KEYWORDS and assumes that the user has the stable keyword selected there. This probably still needs fixing.

Conclusion

It is not too difficult to map the full power of package installations on Gentoo into the puppet way of installing packages. I'm pretty certain that some of the methods I implemented in os_gentoo are still bound to evolve and do not yet represent the best way of handling installations on Gentoo. The module does for example not solve any of the issues mentioned on the Gentoo page in the puppet wiki. So there is still work to be done.

But for now I'm happy to have the central aspects of use flags, keywords and masking available within puppet.

A first positive experience with ruby: Patching puppet

2008-05-15T11:14:00.003+02:00

So far I didn't have much experience with ruby. The few lines of code I've written in that language reminded me too much of perl. And I'm not really a fan of the perl syntax. But today ruby managed to convince me in the area of unit testing.

The problem

I'm bound to stick to ruby as I decided that ruby-based puppet will provide a central element of the next Kolab2/Gentoo version. While it provides some nice LDAP integration features these are not quite sufficient for Kolab. Puppet can grab some host parameters from LDAP and integrate these into the host configuration. The problem for Kolab2/Gentoo is the limitation to some LDAP parameters. Actually these have to be real LDAP attributes that have been defined in a schema.

As I have already argued on the Kolab mailing list it does not make much sense to define attributes in a schema if you want to use such parameters for configuration of a large set of possible applications (postfic, openldap, cyrus, ...). In this case it makes more sense to use the approach also used by the Horde LDAP schema: specifying a single attribute that uses a string value to specify parameters with arbitrary names. E.g. ldapAttribute:"one=two" in order to define parameter "one". Only the "ldapAttribute" will have to be defined in a schema while the code using this parameter handles converting the string into the final paramter.

I wrote a short patch for puppet to implement this. After a short while I got a positive response but the patch was considered insufficient as it lacked any tests.

A simple solution

I admit I was slightly worried because learning to handle just another test framework in a language I have nearly no clue about was something I did not fancy at all. And that was the first really positive surprise about ruby: Using the test framework and successfully writing unit tests in it was a matter of half an hour. Even though it required mocking the LDAP connection.

The testing allowed me to reconsider my expectations concerning the patch and to fix a problem of my initial version. I submitted the new version shortly afterwards and hope it will find its way into the repository now.

Well done, ruby. Let me see what else you can do in order to convince me that you are indeed a good thing...

app-admin/pardalys was created in the Kolab overlay

2008-05-14T22:03:00.005+02:00

If you look at the current ebuild you might wonder what the fuss might be about... It is a pretty empty package.

But the p@rdalys project will form the core for Kolab2/Gentoo-2.2. It will certainly replace net-mail/kolabd and might include some other packages, too.

The idea is to allow you to install Kolab2/Gentoo-2.2 with two simple steps:

emerge app-admin/pardalys
pardalys

Of course there is still a certain way to go until it will actually work that way. And this easy setup is actually just meant as a nice side effect and is not the main point of starting the project. I'll start explaining this package in greater detail once I push more code into it.

For now the link to the project page will be all I can provide.

Currently it might not be clear what the package will actually be about but if people wish to contribute to the project at a later time point you should go visit the git repository on GitHub. This git repository should serve as a scratch repository used for easy sharing and patching of the code. The reference repository on the other hand will be kept in subversion on SourceForge and will be used for packaging.

More on the whole story once there is more code.

The OpenSourceSchool opens its doors

2008-04-25T10:01:00.007+02:00

My publisher started with his next endeavor in bringing knowledge to the masses: The OpenSourceSchool. This time it is about spoken words - or courses - rather than written pages bound as books. Many OpenSourcePress authors are offering seminars there.

I would definitely have liked to offer a course about Gentoo there. But I had to agree with them that this would probably not raise enough interest from paying customers. Or am I wrong about that?

But of course there was room for the second topic dear to my heart: Kolab. The course will take five days and touch all major topics of the Kolab Server. Central components such as postfix, openldap, cyrus imap will provide the core components but I'll certainly also include a chapter about getting the Horde web client successfully installed. So we will hopefully have a new batch of Kolab experts in October.

And hopefully the preparations for the course will also help in laying the groundwork for a book about Kolab. This is the only book I still want to write after going through the pain of writing the Gentoo book.

Polymeraze: Time for merging

2007-12-19T18:00:00.001+01:00

Polymeraze is a tool that I wrote about two years ago. This package allowed me to cope with more than ten Gentoo servers at the same time. One of the main problems with many machines is to keep up with the configuration of the machines.

Polymeraze allows you to create kind of knowledge base for configurations that can be applied to multiple hosts. It basically uses some template configurations which are being converted into the final configuration files by combining them with the specific host profile.

While this tool was extremely handy it was also a bad hack since I wired it directly into the Gentoo package management system. This basically never really worked but since I knew how to handle it I just didn't mind too much.

A while ago I realized that this tool has a few common properties with the good old "kolabconf" from the Kolab project. And since then I have dreamed of replacing kolabconf with a rewritten polymeraze.

I hope that this merge will finally lead to a tool that will allow to get a really solid Kolab server natively integrated into the Gentoo system.

Well, there is still plenty of work ahead. All I did so far is getting the project page up. And now it is time for coding...

K2G News: Update of the project site

2007-12-06T10:08:00.000+01:00

The project site got a few updates and new links. In addition a Ohloh project summary has been created.

K2G: Kolab2/Gentoo getting attention again

2007-12-06T08:56:00.000+01:00

Finally the Kolab2Gentoo project gets some attention again. I started fixing some older bugs but the main part will be to prepare for 2.2 now. I still hate the way the whole project is structured but there are too many places where massive improvements are needed. So I guess I have to be content with small progress. The stuff that is currently on my mind concerning Kolab2/Gentoo-2.2:

Update to Horde-3.2
Making the patched c-client and php packages recommended instead of required
Switching to a specific cyrus-imapd-kolab package
Template packages

And then there is the big, bad problem of the Kolab configuration concept that simply fails for Gentoo in its current form. The solution to that one will take even longer.