Managing Complex rsyslog Configs

If you’re a systems administrator and only concerned with a local system’s logging, you’re familiar with /etc/rsyslog.conf, and that’s all you probably need to be familiar with.  If your aim is to provide rsyslog as a service to broad base of diverse users, you’re going to quickly outgrow /etc/rsyslog.conf.  You’re going to have multiple inputs and outputs, a laundry list of rules and templates, and you’ll need keep control of the configuration for the sake of your own sanity.  In this post I’ll go over a few of the methods and tools that I use to keep a large rsyslog service easy to configure, easy to maintain, and running smoothly.

First and foremost – ditch any notion you have of a unified configuration in a single file.  Yes, tools like Ansible and Puppet can easily generate and manage these long monolithic config files for you, but they can just as easily manage multiple configuration files as well.  Spreading your configuration in to multiple files makes things easy to find and more importantly easy to troubleshoot.

Since you’re going to have a number of configuration files, you’ll need to organize them.  My recommendation is a simple directory structure by type of configuration element.  I use the following basic structure (descending from my rsyslog installation location):

etc/rsyslog.d/globals
etc/rsyslog.d/modules
etc/rsyslog.d/templates
etc/rsyslog.d/inputs
etc/rsyslog.d/rules

Then, using $IncludeConfig, your main rsyslogd.conf only needs to contain:

$IncludeConfig etc/rsyslog.d/globals/*.conf
$IncludeConfig etc/rsyslog.d/modules/*.conf
$IncludeConfig etc/rsyslog.d/templates/*.conf
$IncludeConfig etc/rsyslog.d/inputs/*.conf
$IncludeConfig etc/rsyslog.d/rules/*.conf

Then within those directories all configuration files are prefixed with a two digit number, i.e.:

rules/01_udprules.conf
rules/02_tcprules.conf
rules/03_relprules.conf

Prefixing like this has a few benefits:

  • It gives you control over the order in which rsyslog parses and evaluates its configuration.  In day-to-day operations, the order in which the configuration is loaded isn’t critical, but if you ever have to run rsyslog in debug mode (and you will) this will make a huge difference in making the debug logs more readable and grep’able.
  • You can easily associate similar configuration elements across multiple files and directories.  For example if you prefix all of your UDP-related configurations with 01_, a simple find can return all your UDP configurations:
# find . -name "01_"
rsyslog.d/modules/01_imudp.conf
rsyslog.d/inputs/01_udpinputs.conf
rsyslog.d/templates/01_file.conf
rsyslog.d/templates/01_elasticsearch.conf
rsyslog.d/rules/01_udp_to_file.conf
rsyslog.d/rules/01_udp_to_elasticsearch.conf
#

Using this method of subdirectories and $IncludeConfig also scales very well.  Over time you might find the number of template files growing out of control.  Just whip up a few more directories under templates/ to sort them:

templates/sales
templates/ist
templates/engineering

Then drop a config in the top-level templates/ with:

$IncludeConfig sales/*.conf
$IncludeConfig ist/*.conf
$IncludeConfig engineering/*.conf

And you’re set

This method also allows you to quickly update, add and delete configuration elements with minimal risk.  Need another input?  Just drop another .conf file in the inputs directory and reload.  And if you fat-finger that new config and it doesn’t work, the chances of that affecting any other working element are minimized.

Side note – $IncludeConfig is an often overlooked and versatile configuration tool for rsyslog, it is your friend.

See this repo for fully detailed example of this structure, along with some sample configs

 

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s