If you’re a systems administrator and only concerned with a local system’s logging, you’re familiar with /etc/rsyslog.conf, and that’s all you probably need to be familiar with. If your aim is to provide rsyslog as a service to broad base of diverse users, you’re going to quickly outgrow /etc/rsyslog.conf. You’re going to have multiple inputs and outputs, a laundry list of rules and templates, and you’ll need keep control of the configuration for the sake of your own sanity. In this post I’ll go over a few of the methods and tools that I use to keep a large rsyslog service easy to configure, easy to maintain, and running smoothly.
First and foremost – ditch any notion you have of a unified configuration in a single file. Yes, tools like Ansible and Puppet can easily generate and manage these long monolithic config files for you, but they can just as easily manage multiple configuration files as well. Spreading your configuration in to multiple files makes things easy to find and more importantly easy to troubleshoot.
Since you’re going to have a number of configuration files, you’ll need to organize them. My recommendation is a simple directory structure by type of configuration element. I use the following basic structure (descending from my rsyslog installation location):
etc/rsyslog.d/globals etc/rsyslog.d/modules etc/rsyslog.d/templates etc/rsyslog.d/inputs etc/rsyslog.d/rules
Then, using $IncludeConfig, your main rsyslogd.conf only needs to contain:
$IncludeConfig etc/rsyslog.d/globals/*.conf $IncludeConfig etc/rsyslog.d/modules/*.conf $IncludeConfig etc/rsyslog.d/templates/*.conf $IncludeConfig etc/rsyslog.d/inputs/*.conf $IncludeConfig etc/rsyslog.d/rules/*.conf
Then within those directories all configuration files are prefixed with a two digit number, i.e.:
rules/01_udprules.conf rules/02_tcprules.conf rules/03_relprules.conf
Prefixing like this has a few benefits:
- It gives you control over the order in which rsyslog parses and evaluates its configuration. In day-to-day operations, the order in which the configuration is loaded isn’t critical, but if you ever have to run rsyslog in debug mode (and you will) this will make a huge difference in making the debug logs more readable and grep’able.
- You can easily associate similar configuration elements across multiple files and directories. For example if you prefix all of your UDP-related configurations with 01_, a simple find can return all your UDP configurations:
# find . -name "01_" rsyslog.d/modules/01_imudp.conf rsyslog.d/inputs/01_udpinputs.conf rsyslog.d/templates/01_file.conf rsyslog.d/templates/01_elasticsearch.conf rsyslog.d/rules/01_udp_to_file.conf rsyslog.d/rules/01_udp_to_elasticsearch.conf #
Using this method of subdirectories and $IncludeConfig also scales very well. Over time you might find the number of template files growing out of control. Just whip up a few more directories under templates/ to sort them:
templates/sales templates/ist templates/engineering
Then drop a config in the top-level templates/ with:
$IncludeConfig sales/*.conf $IncludeConfig ist/*.conf $IncludeConfig engineering/*.conf
And you’re set
This method also allows you to quickly update, add and delete configuration elements with minimal risk. Need another input? Just drop another .conf file in the inputs directory and reload. And if you fat-finger that new config and it doesn’t work, the chances of that affecting any other working element are minimized.
Side note – $IncludeConfig is an often overlooked and versatile configuration tool for rsyslog, it is your friend.
See this repo for fully detailed example of this structure, along with some sample configs