If you’re a systems administrator and only concerned with a local system’s logging, you’re familiar with /etc/rsyslog.conf, and that’s all you probably need to be familiar with. If your aim is to provide rsyslog as a service to broad base of diverse users, you’re going to quickly outgrow /etc/rsyslog.conf. You’re going to have multiple inputs and outputs, a laundry list of rules and templates, and you’ll need keep control of the configuration for the sake of your own sanity. In this post I’ll go over a few of the methods and tools that I use to keep a large rsyslog service easy to configure, easy to maintain, and running smoothly.
For this post I’m going to walk through how to use Kafka Connect (and the kafka-connect-splunk connector) along with a Splunk Heavy Forwarder to stream data from a Kafka cluster to one or many Splunk instances. If you’re facing the prospect of building a data pipeline to consume data from a Kafka instance and forward it on to Splunk, your list of options is fairly short.
If you want to use rsyslog to reformat syslog data in to JSON format before sending off to an output, you will need to use a template. The template that comes up the most for this purpose is this one:
If you have a large rsyslog environment with multiple inputs, multiple modules, lists of templates and a slew of rules and rulesets, you’ll want to keep it all organized and accessible. Here’s a few pointers for using $IncludeConfig to keep your configuration as simple, easy to maintain and efficient as possible.
If you’re running rsyslog and want to run it securely, you’re going to want to disable SSLv2 and SSLv3. If you’re running rsyslog in a PCI environment you’ll HAVE to. Given recent PCI standards you’ll also have to disable TLS 1.0 (and likely 1.1 in the near future).
First thing you’ll need to do is familiarize yourself with GnuTLS priority strings, and what string fits your needs. Go here: GnuTLS Priority Strings for a full read on priority strings.