Easily Deploy and Configure splunkforwarders via RPM

Introduction

At Aplura, we have many clients that are big Linux shops. These clients have dozens, hundreds, or even thousands of Red Hat Enterprise Linux or CentOS systems that they must manage, configure, and maintain. In our line of work, we deal with small (<50GB) Splunk environments, large (multi-TB) Splunk environments, and everything in between. All of these environments have their own, often unique, ways of managing their Linux systems. One of the challenges for any IT organization is how to handle software installations.

In the case of Splunk Universal Forwarder(UF) installations, we have a unique challenge. Not only do we need to install the splunkforwarder package, we also must configure the UF to contact the Deployment Server and receive additional configurations through Splunk apps.

With Windows, system administrators can use MSIEXEC along with GPOs to distribute the Splunk Universal Forwarder and setup the initial Deployment Server configuration at installation time. Unfortunately, we don’t have the same, built-in capability when installing the Splunk Universal Forwarder via a RPM. However, as with most tasks in the Linux world, we have several ways to skin this cat. Some may choose to script an install. Others will use their favorite Configuration Management system (Chef, Puppet, Salt, Ansible, RDist, …​). These are all viable methods to get your Universal Forwarders out there, especially if you already have the knowledge and infrastructure to do so in your environment.

In this article, we will introduce another method of deploying and configuring Splunk Universal Forwarders on RPM-based Linux systems. We will be creating a site-specific RPM that will allow us to install and configure the Splunk Universal Forwarder in one command. The system will be configured and ready to receive additional configurations via the Deployment Server in your environment. (You do have a Deployment Server, correct?) The site-specific RPM (splunkforwarder-deployment-config) will install the splunkforwarder RPM from Splunk, Inc. as a pre-requisite dependency, create a custom deploymentclient.conf, configure the Splunk Universal Forwarder to start at boot as the user of your choice and accept the license on your behalf.

If you choose, you can install the splunkforwarder-deployment-config and the splunkforwarder RPMs manually. With a little more effort, you can set up a YUM repo (all you need is a web server) then install and configure the splunkforwarder at system-build time via kickstart. If you add the splunkforwarder-deployment-config RPM to your RHN Satellite or Spacewalk system, you can push out and configure the Splunk Universal Forwarder remotely to any system in your environment. It’s pretty cool!

Setup RPM Build Environment

Before we get to the fun stuff, we need to set up a RPM build environment. If you already have a build environment ready to go, you can skip down to the Create Your RPM section. Otherwise, keep reading and we will quickly set up your environment.

For more details on setting up your RPM Build environment, see https://wiki.centos.org/HowTos/SetupRpmBuildEnvironment.

First, we will install the pre-requisite software needed to build our RPM. To install with elevated privileges, run:

sudo yum install rpm-build rpm-sign make git

Now, as a regular user, we will run the following to create the build directory structure in our home directory:

mkdir -p ~/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}

Next, we are going to create a .rpmmacros file which will be used to define several defaults for the RPM build system. To create the .rpmmacros file and use the build directory structure that we created above, run the following:

echo '%_topdir %(echo $HOME)/rpmbuild' >>~/.rpmmacros

We also want to document who created the RPM package. To do so, add your name and email address to the .rpmmacros file. Be sure to substitute and with the appropriate information.

export NAME="<your full name>"
export EMAIL=<your email address>
echo "%packager ${NAME} <${EMAIL}>" >>~/.rpmmacros

Create GPG Keys

In order to maintain the integrity of our systems, we need to ensure that only legitimate, unmodified RPMs are installed on our systems. We need to sign our RPMs with a GPG key so that they can be validated. Run the following command and follow the prompts to create a unique GPG key for RPM signing:

gpg --gen-key

Now, run the following to define the newly created key as the default signing key in the ~/.rpmmacros file:

cat <END >>~/.rpmmacros
#gpg signing
%_signature gpg
%_gpg_name <Replace with User-Id>
END

The User-Id will be a combination of the name, email address and comment that you entered when creating your GPG key. Run gpg --list-keys to find your User-Id and include it in the cat command above. {. notice}

Create Your RPM

With our build environment in place, we can start building the custom splunkforwarder-deployment-config RPM. First, clone or download the git repo from the Aplura Github repostiory:

git clone https://github.com/aplura/splunkforwarder-deployment-config.git

Now, copy the contents of the SPECS directory from the cloned repo to the SPECS directory in the RPM Build environment. Likewise, copy the contents of the SOURCES directory from the cloned repo to the corresponding directory in the RPM Build environment.

[bash gutter="false"]
cd splunkforwarder-deployment-config
cp -R SPECS/* ~/rpmbuild/SPECS/
cp -R SOURCES/* ~/rpmbuild/SOURCES/

Be sure to check out the README.md file for additional information on the customization options available when building your site-specific RPM. At a minimum, we will need the following:

• CLIENTNAME

• DEPLOYMENTSERVER

As indicated in the README file, the CLIENTNAME is not the hostname of the client on which the RPM will be installed. It is a generic name that is used in the Deployment Server configuration to identify what client settings to use.

With the CLIENTNAME, DEPLOYMENTSERVER, and any other custom options determined, we are now ready to build a custom splunkforwarder-deployment-config RPM:

cd ~/rpmbuild/SPECS/
export CLIENTNAME=<Your Deployment Client Name>
export DEPLOYMENTSERVER=<Your deployment server>
export OTHEROPTIONS=<optional Define Statements>
rpmbuild -ba --sign --define "CLIENTNAME ${CLIENTNAME}" --define "DEPLOYMENTSERVER ${DEPLOYMENTSERVER}" ${OTHEROPTIONS} splunkforwarder-deployment-config.spec

Once this command completes without any errors, the new RPM will be ready and waiting in the ~/rpmbuild/RPMS directory.

Install the RPM

At this point we are ready to install our RPM. As mentioned earlier, the splunkforwarder is required in order to install the splunkforwarder-deployment-config RPM. Download the latest splunkforwarder RPM from https://splunk.com, then run the following to install and configure the system as a Deployment Server client:

rpm -Uvh splunkforwarder*rpm splunkforwarder-deployment-config*rpm

After a short time, typically within a minute or so, the splunkforwarder will check into the Deployment Server and receive the appropriate Splunk apps.

Add RPM to an Internal YUM Repo

While installing the splunkforwarder-deployment-config RPM manually is nice and saves a few steps, we still have to copy over the RPMs to each Linux system. That is not convenient and takes too much time in larger environments. To get around this issue, we can add our custom RPM and the splunkforwarder RPM to a local repo. If you don’t already run a local YUM repo, place the RPMs in an web-accessible directory on a suitable server and use the createrepo command to build the repo.

Run `yum install createrepo` to install the `createrepo` executable.

You will need to configure your Enterprise Linux clients to use the custom repo for any RPMs that you want to make accessible, including our splunkforwarder-deployment-config RPM. Then, run the following to install and configure your Splunk Universal Forwarder:

yum install splunkforwarder-deployment-config

Use in Kickstart

Now that we have a local YUM repo with our custom RPM, we can incorporate the splunkforwarder-deployment-config RPM into our Enterprise Linux build process using kickstart. With a custom kickstart configuration, we can install and configure the Splunk Universal Forwarder at system build time rather than after the system has already been built. If we include the splunkforwarder and splunkforwarder-deployment-config RPMs in the local repo, we can automatically install and configure the Splunk Universal Forwarder by adding the following to the %packages section of a kickstart file:

splunkforwarder-deployment-config

As the system is building, the splunkforwarder RPM will be installed as a pre-requisite of the splunkforwarder-deployment-config RPM and the newly created system will be properly configured as a Deployment Server client.

Refer to Red Hat’s page on Kickstart Installations for more information.

Use in RHN Satellite or Spacewalk

To take it a step further, we can also add the splunkforwarder and splunkforwarder-deployment-config RPMs to a RHN Satellite or Spacewalk server. Once added, we can push out the RPMs to any system from a central management system. This will give us the flexibility to start collecting data from any RPM-based system (or group of systems) whenever we want. It’s pretty powerful and will make the life of a Enterprise Linux system administrator that much easier.

Conclusion

Hopefully, this toolkit will be useful and help you better manage your Splunk Universal Forwarders on Enterprise Linux. I encourage anyone who is interested to try out the splunkforwarder-deployment-config toolkit and provide any feedback or contributions to make this project more useful to the Splunk community.