Aplura, LLC.

Aplura, LLC.

Success with rsyslog

6 minute read

For a while now, I’ve been hearing complaints about rsyslog’s configuration format. syslogd style configuration syntax has a reputation for being difficult to read. Understandably, this has caused a preference for syslog-ng with some going as far as ripping out rsyslogd, the default syslog implementation, and replacing it with syslog-ng. In this post, I hope to show that this level of effort is not always needed and rsyslog’s configuration file can be made tolerable.

I set out to do some experimenting with what’s possible in rsyslog’s configuration on RHEL / CentOS 7. My goal was pretty simple – create a template that emulates in rsyslog what I have working in syslog-ng. No more syslogd style filters in this configuration template.

Methodology

My strategy for this template was to break it down into four different functions:

  • Filters – Incoming data routing to various rulesets

  • Destinations – Output formatting for rulesets

  • Rulesets – Utilize destinations here to log to a file

  • Sources – Data inputs that get bound to filters

Technically, the Rulesets option above is redundant. Everything that happens within the “rulesets” function could be moved to the “filters” section. I do find that the additional layer of abstraction enhances readability and maintainability. In large environments, each of the functions listed above could be broken into different files and merged with an import, adding additional maintainability improvements.

Additionally, you can utilize the “ruleset” section to take advantage of some nifty rsyslog features:

  • Queues

    • Rsyslog Queues allow you to build memory or disk backed queues to buffer output. For example, if you’re sending data to a third party system you could use a queue to buffer output in the event of an outage. See the documentation on Understanding rsyslog Queues for more information.

  • Worker threads

    • Worker threads allow you to make use of more CPU to parse and store messages. See the documentation about Worker Thread Pools for more information.

  • Forwarding to another system

    • While this does not require the use of a ruleset, you can utilize one to apply specific formatting or other options before sending along to a third party destination.

I’ll break the config file into sections here. If interested, you can find the latest version of the entire configuration file in my Github repository.

Configuration Sections

Global


     # Global
      # preserve sending host fqdn
      $PreserveFQDN on
      $CreateDirs on

     # Lets log about our queue usage.
      module(
      load="impstats"
      interval="10"
      log.file="/var/log/rsyslog_stats.log"
      log.syslog="off"
      )

     # Receive outside syslog
      # Provides TCP syslog reception
      module(load="imtcp" MaxSessions="5000")
      module(load="imudp")

Here we’re configuring some global variables and enabling rsyslog’s stats log. We are preserving the fully qualified domain name(FQDN), enabled rsyslog to create directories, and allow rsyslog to listen on the default TCP and UDP ports.

Filters


     # Filters
      # Lets setup a queue
      ruleset(name="f_all" queue.type="LinkedList" queue.workerthreads="4" queue.size="100000") {
     # f_network_devices
     if ($fromhost-ip startswith '192.168.100.' or
     $fromhost-ip startswith '192.168.101.' or
     $fromhost-ip startswith '192.168.102.') then {
     call r_network
     stop
     }

     # f_cron
      if ($syslogtag startswith 'cron') then {
      call r_cron
      stop
      }

     # f_kern
      if ($syslogtag startswith 'kern') then {
      call r_kern
      stop
      }


     # f_mail


if ($syslogtag startswith 'mail' or $syslogtag startswith 'sendmail' or $syslogtag startswith 'sm-mta' or $syslogtag startswith 'postfix' or $syslogtag startswith 'sSMTP' ) then { call r_mail stop }

     # f_secure
if ($syslogtag startswith 'sudo' or $syslogtag startswith 'su' or $syslogtag startswith 'dzdo' or $syslogtag startswith 'adclient' or $syslogtag startswith 'runmappers' or $syslogtag startswith 'ssh' or $syslogtag startswith 'passwd' or $syslogtag startswith 'auth' or $syslogtag startswith 'security' or $syslogtag startswith 'login' or $syslogtag startswith 'unix_chkpwd' or $syslogtag startswith 'chage' ) then { call r_secure stop }
\

     # f_puppet
      if ($syslogtag startswith 'puppet') then {
      call r_puppet
      stop
      }

     # f_dhcpd
      if ($syslogtag startswith 'dhcpd') then {
      call r_dhcpd
      stop
      }

     # f_named
      if ($syslogtag startswith 'named') then {
      call r_named
      stop
      }

     # Cisco
      ## acs
      if ($syslogtag contains 'CisACS' or $msg contains 'CisACS') then {
      call r_acs
      stop
      }
      ## pix
      if ($syslogtag contains 'PIX-' or $msg contains 'PIX-') then {
      call r_pix
      stop
      }
      ## asa
      if ($syslogtag contains 'ASA-' or $msg contains 'ASA-') then {
      call r_asa
      stop
      }
      ## cisco
      if ($msg contains '%SEC-' or
      $msg contains '%OSPF-' or
      $msg contains '%LINK-' or
      $msg contains '%SW_MATM-' or
      $msg contains '%IP-' or
      $msg contains '%IP_SNMP-' or
      $msg contains '%C4K_L2MAN-' or
      $msg contains '%CLEAR-' or
      $msg contains "%STACKMGR-" or
      $msg contains "%EVENT-" or
      $msg contains "%EC-" or
      $msg contains "%HA_EM-" or
      $msg contains "%NAC-" or
      $msg contains "%GENERAL" or
      $msg contains "%SNMP-" or
      $msg contains "%CDP-" or
      $msg contains "%ISDN-" or
      $msg contains "%FAN-" or
      $msg contains "%LINE-" or
      $msg contains "%SPANTREE-" or
      $msg contains "%PM-" or
      $msg contains "%SSH-" or
      $msg contains "%LINEPROTO-" or
      $msg contains "%RTD-" or
      $msg contains "%SYS-" or
      $msg contains "srw-recnet" or
      $msg contains "efo-recnet" or
      $msg contains "elw-recnet" or
      $msg contains "sre-recnet" or
      $msg contains "uca-recnet"
      ) then {
      call r_cisco
      stop
      }

     call r_catch_all
      stop
      }

We setup a lot of filters there. The main point is to show off the flexibility of filtering with rulesets in rsyslogd. If you need more details or want to dive deeper into rsyslog filters see the official documentation on rsyslog filters.

I don’t necessarily recommend splitting incoming data out quite so verbosely. That said, there are plenty of use cases where filtering by an IP range or a program name would come in handy. For example, splitting out Cisco IOS data by program name will make storing this data in it’s own log file significantly easier.

The big takeaway here is the “call” syntax that enables us to chain rulesets together. In syslog-ng an action would be setup with a log statement. With rsyslog, the order of operations is slightly different. An action is called from a filter. By utilizing another ruleset as the action, we are also able to either add additional filtering or use a dedicated queue and set of worker threads.

Destinations



     # Destinations
      #
      template(name="d_catch_all" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%.log")

     # Network
      template(name="d_network" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%.log")

     # *nix
      template(name="d_cron" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%-cron.log")
      template(name="d_kern" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%-kernel.log")
      template(name="d_mail" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%-mail.log")
      template(name="d_secure" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%-auth.log")
      template(name="d_puppet" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%-puppet.log")
      template(name="d_dhcpd" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%-dhcpd.log")
      template(name="d_named" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%-named.log")
      template(name="d_unix_other" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%-unix_other.log")

     # Cisco
      template(name="d_acs" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%-cisco_acs.log")
      template(name="d_pix" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%-asa_pix.log")
      template(name="d_asa" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%-cisco_asa.log")
      template(name="d_cisco" type="string" string="/var/log/remote_syslog/%FROMHOST%/%$YEAR-%$MONTH%-%$DAY%-cisco.log")

Here we provide templates to be used later on for output formats. In these examples, we are separating the log files by source host then year, month, and day.

Rulesets


     # rulesets

     # Network
      ruleset(name="r_network"){
      action(type="omfile" DynaFile="d_network")
      }

     # *nix
      ruleset(name="r_cron"){
      action(type="omfile" DynaFile="d_cron")
      }
      ruleset(name="r_kern"){
      action(type="omfile" DynaFile="d_kern")
      }
      ruleset(name="r_mail"){
      action(type="omfile" DynaFile="d_mail")
      }
      ruleset(name="r_secure"){
      action(type="omfile" DynaFile="d_secure")
      }
      ruleset(name="r_puppet"){
      action(type="omfile" DynaFile="d_puppet")
      }
      ruleset(name="r_dhcpd"){
      action(type="omfile" DynaFile="d_dhcpd")
      }
      ruleset(name="r_named"){
      action(type="omfile" DynaFile="d_named")
      }
      ruleset(name="r_unix_other"){
      action(type="omfile" DynaFile="d_unix_other")
      }

     # Cisco
      ruleset(name="r_acs"){
      action(type="omfile" DynaFile="d_acs")
      }
      ruleset(name="r_pix"){
      action(type="omfile" DynaFile="d_pix")
      }
      ruleset(name="r_asa"){
      action(type="omfile" DynaFile="d_asa")
      }
      ruleset(name="r_cisco"){
      action(type="omfile" DynaFile="d_cisco")
      }

     # catch_all
      ruleset(name="r_catch_all"){
      action(type="omfile" DynaFile="d_catch_all")
      }

This is the “action” section of the config. We declare actions to be taken for the filters above. These rulesets are called from the filter section and reference the destinations declared above.

Sources


     # Sources
      input(type="imtcp" port="514" ruleset="f_all")
      input(type="imudp" port="514" ruleset="f_all")

      input(type="imtcp" port="1540" ruleset="f_all")
      input(type="imudp" port="1540" ruleset="f_all")

Pretty basic source definitions here. We configure TCP and UDP inputs and bind them to a ruleset. Binding them to unique rulesets is also an option. A handy trick to use is to send all data of a specific type to a unique port. This would allow for skipping the filter section entirely and just calling an output rule.

I’ve uploaded two configs to a github repository, a complex template and a simple template. The outline described in this post was from the complex template. You can also check out our syslog cheat sheet, which covers both rsyslog and syslog-ng.

As always, any constructive feedback on the templates is appreciated!

Updated:

You May Also Enjoy

9/23/2019 Baltimore Splunk User Group

less than 1 minute read

September meeting! Due to conflicts, this meeting is on Sep 23. We will meet in Towson, and I’ll practice my .conf talk (Lesser Known) ! Also food and drinks...