Splunk, Disney, and You

Splunk .conf 2016 held at Walt Disney World is in the books, and it was a great week! I was fortunate enough to be able to bring my family along this year, something I wouldn’t have considered if Splunk .conf was in Las Vegas as in years past. My family and I loved being at Disney. Throughout the week I was continually noticing the very high attention to detail that is present in the Disney experience. From the seamlessness of the travel to and from the airport and resort, the usefulness of the Disney app on my phone, to things like the timing of the fireworks with the musical presentations.

This might sound like a tangent and not really related to Splunk, but you can learn some important lessons while enjoying the magical experience. These lessons can inform you and help you create great experience for your Splunk users.

It’s All in the Details

First and foremost is Disney’s fanatical attention to detail.

On our third day at WDW, we visited Epcot and watched Illuminations. This is when I became aware of the high level of execution in the Disney experience. For some quick background, my first job out of high school was with a company that installed sound systems in arenas and skating rinks. While in that job I learned about the challenges involved in creating sound systems for large open spaces. During the Illuminations show, I noticed that the fireworks were in near perfect sync with the music. If you’re not familiar with it, Illuminations is a show that combines fireworks, lasers, and very large globe with images projected on it’s surface that is all timed with music. The timing is so good that the different aspects of the performance all blend together to become a complete experience. This seamlessness is very difficult to accomplish, and most people wouldn’t really notice if the fireworks were a little off from the music. It would still be a pretty good show. Disney’s engineers and producers really worked on the details, like the timing of the fireworks explosions, and these small things add up to elevate the experience from “pretty good” to magical.

Splunk’s Magic

As I stood there with my family enjoying the experience, I was also thinking back across our first three days at Walt Disney World and realizing that this wasn’t the first seamless experience we’d encountered. We started planning months before this trip, and throughout the process the Disney attention to detail was at work. Scheduling park visits, scheduling rides with FastPass, getting our luggage tags and Magic Bands was easier than I expected. The whole process was crafted to let us know about and decide on important details and not worry about the unimportant details. This type of experience is your goal as a Splunk admin: Create a Splunk environment that allows users to answer questions and solve problems easily. Users should be able to make relevant choices and not be slowed by unimportant details.

Learn the Magic

Splunk’s configuration is full of small details that can make your administrative effort more effective and help you create a more consistent experience for you and your users. Often, the default settings are the safest option, but not the most effective option. You should understand the settings that will affect your users and configure them appropriately.

• Splunk’s documentation is excellent and should be your primary reference when configuring Splunk.

• Splunk Answers is a great place for finding solutions to problems. Get in the habit of checking Answers when you are having a problem.

• Splunk Usergroups are a great place to find like-minded people who are passionate about Splunk technology. Be sure to sign up for the Usergroup Slack Chat.

Configure the Magic

Here are some things you can do to help you get the most out of your Splunk configs:

• Create a set of deployment apps that take advantage of Splunk’s layered configuration files to reduce the amount of manual configuration needed to administer Splunk.

• Create an outputs app to distribute your outputs.conf to all of your forwarders and search heads. Adding or changing indexers or intermediate forwarders becomes a much simpler process than manually changing each forwarder.

• Create an authentication app for authorize.conf and authentication.conf and distribute it to your search heads. These config files contain LDAP configurations and LDAP group to role mappings. This allow you to maintain a consistent login across all search heads, reduce the possibility of unauthorized access to sensitive data, and allow for easy auditing of user activity.

Magic Band FTW

Weeks before we traveled to Disney, my family received a box containing our Magic Bands. These are wristbands look very much like an activity/exercise tracker wristband. Even the kids had their own custom-colored bands. Everywhere in Disney World we used our Magic Bands to get access to our room, the parks, and our wallets (ok, maybe there is a downside to that last part…). They also let us access the express lines for rides. My children loved using them to open the door to our room and at the pool (especially the giant water slide).

Use Splunk’s LDAP and Single Sign-On

• Make use of the LDAP integration for Splunk logins and role-based access (RBAC). This reduces the admin overhead for you and simplifies your users’ experience.

• Use SAML authentication for SSO. Splunk 6.5 spports the following identity providers:

  • Ping Identity

  • Okta

  • Azure AD

  • AD FS

  • OneLogin

  • Optimal

  • CA siteminder

Close the Loop

Think about the way Disney directs guests as they move between resorts and parks and between rides and attractions within a park. Clear signage, well-defined paths, and cast members seemingly every ten feet while moving about the parks. The cast members were happy to answer questions, give directions, or even just to say happy birthday to those little princes and princesses wearing a birthday button.

If you are going to make the most of Splunk, you must give your users a consistent experience while using their apps. Give them a path to follow, and make the path clear.

Stay on the Path

• Avoid dumping users into raw events with default drilldowns.

  • If you know where users should go when drilling into dashboard panels, then lead them to where they need to go.

  • If a logical drilldown is not available for a particular panel, disable the drilldown for that panel.

• Use inputs on your dashboards to allow your users to get deeper into the details without dumping them into raw events. By default, the search option is always available from a dropdown, and your users that are comfortable with raw search are free to use it as needed.

The Magical Conclusion

The big lesson I learned from my Splunk .conf 2016 experience is that the details matter, and it’s all in the details. It takes effort and imagination to continually improve your Splunk deployment but that effort will pay off in the form of more effective users and reduced administrative overhead.