Success with rsyslog

Success with rsyslog

-

For a while now, I’ve been hearing complaints about rsyslog’s configuration format. syslogd style configuration syntax has a reputation for being difficult to read. Understandably, this has caused a preference for syslog-ng with some going as far as ripping out rsyslogd, the default syslog implementation, and replacing it with syslog-ng. In this post, I hope to show that this level of effort is not always needed and rsyslog’s configuration file can be made tolerable.

I set out to do some experimenting with what’s possible in rsyslog’s configuration on RHEL / CentOS 7. My goal was pretty simple — create a template that emulates in rsyslog what I have working in syslog-ng. No more syslogd style filters in this configuration template.

Methodology

My strategy for this template was to break it down into four different functions:

  • Filters — Incoming data routing to various rulesets
  • Destinations — Output formatting for rulesets
  • Rulesets — Utilize destinations here to log to a file
  • Sources — Data inputs that get bound to filters

Technically, the Rulesets option above is redundant. Everything that happens within the “rulesets” function could be moved to the “filters” section. I do find that the additional layer of abstraction enhances readability and maintainability. In large environments, each of the functions listed above could be broken into different files and merged with an import, adding additional maintainability improvements.

Additionally, you can utilize the “ruleset” section to take advantage of some nifty rsyslog features:

  • Queues
    • Rsyslog Queues allow you to build memory or disk backed queues to buffer output. For example, if you’re sending data to a third party system you could use a queue to buffer output in the event of an outage. See the documentation on Understanding rsyslog Queues for more information.
  • Worker threads
    • Worker threads allow you to make use of more CPU to parse and store messages. See the documentation about Worker Thread Pools for more information.
  • Forwarding to another system
    • While this does not require the use of a ruleset, you can utilize one to apply specific formatting or other options before sending along to a third party destination.

I’ll break the config file into sections here. If interested, you can find the latest version of the entire configuration file in my Github repository.

Configuration Sections

Global

 

#Global
 # preserve sending host fqdn
 $PreserveFQDN on
 $CreateDirs on
# Lets log about our queue usage.
 module(
 load="impstats"
 interval="10"
 log.file="/var/log/rsyslog_stats.log"
 log.syslog="off"
 )
#Receive outside syslog
 # Provides TCP syslog reception
 module(load="imtcp" MaxSessions="5000")
 module(load="imudp")

Here we’re configuring some global variables and enabling rsyslog’s stats log. We are preserving the fully qualified domain name(FQDN), enabled rsyslog to create directories, and allow rsyslog to listen on the default TCP and UDP ports.

Filters

# Filters
 # Lets setup a queue
 ruleset(name="f_all" queue.type="LinkedList" queue.workerthreads="4" queue.size="100000") {
# f_network_devices
if ($fromhost-ip startswith '192.168.100.' or
$fromhost-ip startswith '192.168.101.' or
$fromhost-ip startswith '192.168.102.') then {
call r_network
stop
}
# f_cron
 if ($syslogtag startswith 'cron') then {
 call r_cron
 stop
 }
# f_kern
 if ($syslogtag startswith 'kern') then {
 call r_kern
 stop
 }
# f_mail

if ($syslogtag startswith ‘mail’ or $syslogtag startswith ‘sendmail’ or $syslogtag startswith ‘sm-mta’ or $syslogtag startswith ‘postfix’ or $syslogtag startswith ‘sSMTP’ ) then { call r_mail stop }

# f_secure

if ($syslogtag startswith ‘sudo’ or $syslogtag startswith ‘su’ or $syslogtag startswith ‘dzdo’ or $syslogtag startswith ‘adclient’ or $syslogtag startswith ‘runmappers’ or $syslogtag startswith ‘ssh’ or $syslogtag startswith ‘passwd’ or $syslogtag startswith ‘auth’ or $syslogtag startswith ‘security’ or $syslogtag startswith ‘login’ or $syslogtag startswith ‘unix_chkpwd’ or $syslogtag startswith ‘chage’ ) then { call r_secure stop }

# f_puppet
 if ($syslogtag startswith 'puppet') then {
 call r_puppet
 stop
 }
# f_dhcpd
 if ($syslogtag startswith 'dhcpd') then {
 call r_dhcpd
 stop
 }
# f_named
 if ($syslogtag startswith 'named') then {
 call r_named
 stop
 }
# Cisco
 ## acs
 if ($syslogtag contains 'CisACS' or $msg contains 'CisACS') then {
 call r_acs
 stop
 }
 ## pix
 if ($syslogtag contains 'PIX-' or $msg contains 'PIX-') then {
 call r_pix
 stop
 }
 ## asa
 if ($syslogtag contains 'ASA-' or $msg contains 'ASA-') then {
 call r_asa
 stop
 }
 ## cisco
 if ($msg contains '%SEC-' or
 $msg contains '%OSPF-' or
 $msg contains '%LINK-' or
 $msg contains '%SW_MATM-' or
 $msg contains '%IP-' or
 $msg contains '%IP_SNMP-' or
 $msg contains '%C4K_L2MAN-' or
 $msg contains '%CLEAR-' or
 $msg contains "%STACKMGR-" or
 $msg contains "%EVENT-" or
 $msg contains "%EC-" or
 $msg contains "%HA_EM-" or
 $msg contains "%NAC-" or
 $msg contains "%GENERAL" or
 $msg contains "%SNMP-" or
 $msg contains "%CDP-" or
 $msg contains "%ISDN-" or
 $msg contains "%FAN-" or
 $msg contains "%LINE-" or
 $msg contains "%SPANTREE-" or
 $msg contains "%PM-" or
 $msg contains "%SSH-" or
 $msg contains "%LINEPROTO-" or
 $msg contains "%RTD-" or
 $msg contains "%SYS-" or
 $msg contains "srw-recnet" or
 $msg contains "efo-recnet" or
 $msg contains "elw-recnet" or
 $msg contains "sre-recnet" or
 $msg contains "uca-recnet"
 ) then {
 call r_cisco
 stop
 }
call r_catch_all
 stop
 }

We setup a lot of filters there. The main point is to show off the flexibility of filtering with rulesets in rsyslogd. If you need more details or want to dive deeper into rsyslog filters see the official documentation on rsyslog filters.

I don’t necessarily recommend splitting incoming data out quite so verbosely. That said, there are plenty of use cases where filtering by an IP range or a program name would come in handy. For example, splitting out Cisco IOS data by program name will make storing this data in it’s own log file significantly easier.

The big takeaway here is the “call” syntax that enables us to chain rulesets together. In syslog-ng an action would be setup with a log statement. With rsyslog, the order of operations is slightly different. An action is called from a filter. By utilizing another ruleset as the action, we are also able to either add additional filtering or use a dedicated queue and set of worker threads.

Destinations

# Destinations
 #
 template(name="d_catch_all" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%.log")
# Network
 template(name="d_network" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%.log")
# *nix
 template(name="d_cron" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%-cron.log")
 template(name="d_kern" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%-kernel.log")
 template(name="d_mail" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%-mail.log")
 template(name="d_secure" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%-auth.log")
 template(name="d_puppet" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%-puppet.log")
 template(name="d_dhcpd" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%-dhcpd.log")
 template(name="d_named" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%-named.log")
 template(name="d_unix_other" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%-unix_other.log")
# Cisco
 template(name="d_acs" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%-cisco_acs.log")
 template(name="d_pix" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%-asa_pix.log")
 template(name="d_asa" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%-cisco_asa.log")
 template(name="d_cisco" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%-cisco.log")

Here we provide templates to be used later on for output formats. In these examples, we are separating the log files by source host then year, month, and day.

Rulesets

# rulesets
 #
# Network
 ruleset(name="r_network"){
 action(type="omfile" DynaFile="d_network")
 }
# *nix
 ruleset(name="r_cron"){
 action(type="omfile" DynaFile="d_cron")
 }
 ruleset(name="r_kern"){
 action(type="omfile" DynaFile="d_kern")
 }
 ruleset(name="r_mail"){
 action(type="omfile" DynaFile="d_mail")
 }
 ruleset(name="r_secure"){
 action(type="omfile" DynaFile="d_secure")
 }
 ruleset(name="r_puppet"){
 action(type="omfile" DynaFile="d_puppet")
 }
 ruleset(name="r_dhcpd"){
 action(type="omfile" DynaFile="d_dhcpd")
 }
 ruleset(name="r_named"){
 action(type="omfile" DynaFile="d_named")
 }
 ruleset(name="r_unix_other"){
 action(type="omfile" DynaFile="d_unix_other")
 }
# Cisco
 ruleset(name="r_acs"){
 action(type="omfile" DynaFile="d_acs")
 }
 ruleset(name="r_pix"){
 action(type="omfile" DynaFile="d_pix")
 }
 ruleset(name="r_asa"){
 action(type="omfile" DynaFile="d_asa")
 }
 ruleset(name="r_cisco"){
 action(type="omfile" DynaFile="d_cisco")
 }
# catch_all
 ruleset(name="r_catch_all"){
 action(type="omfile" DynaFile="d_catch_all")
 }

This is the “action” section of the config. We declare actions to be taken for the filters above. These rulesets are called from the filter section and reference the destinations declared above.

Sources

# Sources
 #
 input(type="imtcp" port="514" ruleset="f_all")
 input(type="imudp" port="514" ruleset="f_all")
 input(type="imtcp" port="1540" ruleset="f_all")
 input(type="imudp" port="1540" ruleset="f_all")

Pretty basic source definitions here. We configure TCP and UDP inputs and bind them to a ruleset. Binding them to unique rulesets is also an option. A handy trick to use is to send all data of a specific type to a unique port. This would allow for skipping the filter section entirely and just calling an output rule.

I’ve uploaded two configs to a github repository, a complex template and a simple template. The outline described in this post was from the complex template. You can also check out our syslog cheat sheet, which covers both rsyslog and syslog-ng.

As always, any constructive feedback on the templates is appreciated!

View All News