Let’s revisit Storage Passwords!

Changes over the years have modified the requirements for the encrypted credentials (/storage/passwords) store, so it’s time to re-visit what is possible!

TL;DR:

• Old versions of Splunk (<=6.4) had a limitation of 255 bytes per credential.
• New versions of Splunk (>=6.5) [since 2016] do NOT have a limit of characters per credential.
• Splunk’s Add-on Builder still implements the old limitation within its configuration.
• Update your credentials using a script to allow for a single stanza compatible with Secure Password Stores!

History

Way back when, in the years of Splunk Enterprise 4.3 (2012), a new feature was introduced called “Storage Passwords”. This feature replaced the old endpoint /admin/passwords and allowed for more flexibility in storing encrypted secrets within Splunk. The “new” endpoint is /storage/passwords, and has been in use for many iterations of the product. However, when implemented initially, there was a limit of 255 bytes per credential. This was presumably part of limitations in the code during that time (it was the wild west of course!). It should be noted during my research, I could not find that limitation documented anywhere.

So what?

Because more recently, that limitation was removed. Being an older creature within the Splunk realm, I maintain backups of older copies of Splunk for research purposes. So, I loaded up multiple versions of Splunk to see when that limitation was removed. The winning version was Splunk 6.5, released in 2016. How did I test this? Very simply, I generated a file with 43,700 bytes of Lorem ipsum and used the curl command to upload to the storage passwords endpoint. I used a script to generate varying lengths of “credentials” and uploaded each difference size to the credential store. Once uploaded, I ran an SPL command (found below in References) that gave me the Splunk version, and the results of the credential store. I repeated for versions 5.0, 6.0, 6.1, 6.2, 6.3, 6.4, and 6.5.

As seen here, the results of the experiment for Splunk 6.4 show that no matter the length of the “credential”, the value was truncated to 255 characters.

Figure 1: Splunk 6.4 Credential Store with truncated credentials

Once I tested Splunk 6.5, I found that indeed the limit was removed and I can store full-fledged strings within the credential store. Once again, I failed to find any documentation on the change from 255 characters to basically unlimited.

Figure 2: Splunk 6.5 Credential Store with full credentials

And So?

Now we come to the crux of the issue. Splunk’s Add-on Builder is used to create various Add-ons and Apps for Splunk. This add-on has a release listed at Feb. 21, 2017 for version 2.1.0, listing compatibility for Splunk 6.4 AND 6.5, which bridges the versions with the limitation removal. The add-on utilizes (currently at least, it is unknown if earlier versions used the same library) the library solnlib to allow for various Splunk integrations. It is in this library that (even as of today [v7.0.0]) the encrypted credential store code splits each credential up into stanzas of 255 characters. With AoB, each credential is split, and then encrypted individually within the passwords.conf file, so it is very difficult to update the credentials via configuration file using actual automation techniques. Therefore a credential with length 1600 would be split into 6 stanzas, with a final stanza to determine “the end of the credential”. The splitting of credentials is causing issues with automated credential rotation and integrations with newer credential management systems due to method in which the credentials are handled. This splitting of credentials is no longer required, and the AoB (and by extension solnlib) should be updated to use a single stanza per credential.

Now What?

Therefore, a script must be used to transform the old-style stanzas into a newer single credential per stanza, and disregard the splitting of the credentials. To help remediate the issue, the python found here can be utilized to combine the multiple stanzas into a single credential that can used with AoB developed credential management. It accepts new passwords over standard in from either a file or from a piped command, as well as inline with the command as the final argument.

Note: It is not required to convert the passwords for AoB developed add-ons. This script should be used only by an authorized user, and when credential management is required without splitting the credential across stanzas.

This script should be run anytime there are multiple stanzas for a single credential in a passwords.conf file, and need to be a singular stanza to handle rotations/updates from a CMS.

Warning: Please TEST the script in non-production. It is DESTRUCTIVE and will remove stanzas from the configuration file. We are not liable for any issues arising from the sample script.

With the warnings out of the way, let’s discuss how this can be used. This script requires the use of the Splunk provided python for specific libraries to find the splunk.secret for encryption and decryption.

On Linux based systems, this is the command to use Splunk’s python.

/opt/splunk/bin/splunk cmd python3

Let’s assume we will be convert credentials for the app IA-generic-app.

cp convert-aob-cred.py /opt/splunk/etc/apps/IA-generic-app/bin
cd /opt/splunk/etc/apps/IA-generic-app/bin

For the actual execution, you will need the stanza name for the credential in question. That can be found by reviewing the passwords.conf file and extracting the information from the stanza. It is also an option to grab it from the command line (Linux based), with varying results depending on the OS in use.

splunk cmd btool passwords list | grep REST_CREDENTIAL__ | awk -F "_#" '{print $2}' | awk -F "\`\`" '{print $1}' | sort | uniq

In our case, the stanza credential name will be IA-generic-app#configs/conf-app_inputs:Sample, which was pulled from a stanza

[credential:__REST_CREDENTIAL__#IA-generic-app#configs/conf-app_inputs:Sample``splunk_cred_sep``1:]

Once the credential name stub is found, execute the conversion on the passwords.conf file.

Warning: Again, this script is DESTRUCTIVE and will remove stanzas from the configuration file. We are not liable for any issues arising from the sample script.

/opt/splunk/bin/splunk cmd python3 ./convert-aob-cred.py ../local/passwords.conf IA-generic-app#configs/conf-app_inputs:Sample

Once it runs, the stanzas will be updated within the passwords.conf file, to have a single stanza for the credential, and a second for the end delimiter to provide compatibility with AoB inputs. It is also possible to update the password from the command line.

Consume StdOut from a command

echo "new_password" | /opt/splunk/bin/splunk cmd python3 convert-aob-cred.py <password_file> <input_name> -

Consume contents of a file

/opt/splunk/bin/splunk cmd python3 convert-aob-cred.py <password_file> <input_name> - < <filename>

Just set via string

/opt/splunk/bin/splunk cmd python3 convert-aob-cred.py <password_file> <input_name> "new_password"

Note: The credential is decrypted/encrypted using the `splunk.secret` of the instance running the script, NOT the final destination instance (if other than the local instance)

And that is it! The resulting updates should still be compatible with existing inputs, and are encrypted on disk. Happy Converting!

References

• Splunk 4.3 was released on January 10, 2012.
• Splunk 4.3 API Documentation
• Splunk Enterprise 6.5 was released on September 27, 2016.
• Add on Builder 2.1.0 was released Feb. 21, 2017.

SPL to verify credential lengths

|rest splunk_server=local /servicesNS/-/-/storage/passwords  
| eval length=len(clear_password) 
| eval clear_password = substr(clear_password,0,50)."..." 
| table title length clear_password 
| append [ rest /services/server/info
    | table version
    | rename version as length 
    | eval title="Splunk Version"
    ] 
| stats values(*) as * by title 
| sort - title 
| table title length clear_password