outputs.conf: sslPassword not hashing if placed under the [tcpout] stanza

[bowendenning]

Hey all,

I've configured $SPLUNK_HOME/etc/system/local/outputs.conf to use SSL certificates for forwarding logs. My problem is that sslPassword is not being automatically hashed by splunk when placed under the [tcpout] stanza:

[tcpout]
defaultGroup = splunkssl
sslCertPath = /opt/splunkforwarder/etc/auth/test-server.pem
sslRootCAPath = /opt/splunkforwarder/etc/auth/test-ca.chain
sslPassword = password
sslVerifyServerCert = true

The above config goes on each forwarder and is unique (because a different private key password has been used for each server). The rest of the configuration (the splunkssl group and the actual indexers to forward logs to) is deployed as an app from the deployment server so it can be easily changed in the future.

This works just fine when I take a password previously hashed password and manually configure sslPassword with it, but putting an unhashed password in the outputs.conf file and restarting Splunk doesn't hash this value (as I believe it should).

Bowen

PS: Using Splunk Universal Forwarder 6.1. Problem exists on both Linux and Windows hosts.

[mckeon]

I have the same issue. The hashed sslPassword gets appended to an existing app which I use for everything but the password, rather than to /local/outputs.conf

Of course, this is not ideal behavior. The whole reason I deployed the outputs.conf to /local was so that the password could get hashed, yet the forwarder went ahead and hashed it in the app, like the documentation states it will NOT do.

[jhinkle]

One thing I have noticed is that if there is an error anywhere in the outputs.conf file, Splunk will not hash the password. Also, contrary to what some have said, you must have the password in plaintext, and let the splunk instance hash the password on startup. Even if you are copying the SSL certs from another forwarder and try to just copy the outputs.conf file, for us at least we needed to re-enter the password again in plaintext.

Might not help you, but I'd be sure to look in the splunkd logs and other logs for errors centered around the outputs.conf file and SSL connection. Banged my head for a day looking for the cause of this only to find that due to a syntax error in my outputs.conf file the parsing failed and the password was never even read.

We are using 6.2.1 btw.

[claudio_manig]

Give that man a cookie!
In my case it hashed all passwords and secrets except the ssl password. Be aware that it has to be an error within the tcpout stanza or something the stanza refers to. In my case we used indexer discovery on the forwarders and under the [indexer_discovery:] stanza, the master_uri had no https in front. So, the [tcpout:] stanza refers to that uri by indexerDiscovery = . However, since we changed it to https in the indexer_discovery stanza it worked and the pw have been hashed.

So at the end: if there is any missconfiguraiton or typo whitin the tcpout stanza or something which refers to it, the ssl pw wont' be hashed!

Thanks a lot, your answer saved a lot of headache.

Cheers

[bowendenning]

I tried using 6.1.1 last week but the same problem is still present. Looks like this has been reported previously here too: http://answers.splunk.com/answers/76718/system-sslpassword-not-hashing

Is anyone on this support fourm able to tell me how I would go about raising this as a bug with Splunk?

Currently I'm having to work around this problem by rewriting my deployment script to place the sslPassword directive under the group stanza, restart splunk, and then move the hashed password back up in to the global stanza where it needs to be. This certainly seems like a bug to me.

Bowen

[bowendenning]

I tried using 6.1.1 last week but the same problem is still present. Looks like this has been reported previously here too: http://answers.splunk.com/answers/76718/system-sslpassword-not-hashing

Is anyone on this support fourm able to tell me how I would go about raising this as a bug with Splunk?

Currently I'm having to work around this problem by rewriting my deployment script to place the sslPassword directive under the group stanza, restart splunk, and then move the hashed password back up in to the global stanza where it needs to be. This certainly seems like a bug to me.

Bowen

[bowendenning]

Hey Cam343,

I can confirm that the password isn't in an App, it's under $SPLUNK_HOME/etc/system/local/outputs.conf on the server from which the logs are being sent. It's not a permissions issue or an issue with splunk.secret because the password works (when hashed), it's just that Splunk isn't hashing the password when it starts if the sslPassword directive is placed under the [tcpout] stanza (but it will still work if you put an already hashed password there).

The example configuration that you posted would work fine, passwords will get hashed under all stanzas except the [tcpout] stanza.

I'm using SUF 6.1.0, so am intending to try 6.1.1 later today. I would consider the behaviour that I'm seeing to be a bug, so if it's still a problem with the latest release of the SUF then I'll see if I can figure out how to report it to Splunk.

Bowen

[bowendenning]

SUF 6.1.1 has the same problem. I can't see any reason why Splunk wouldn't hash the sslPassword directive when it's placed under the [tcpout] stanza. Like I said, placing the hashed value there works just fine.

I've posted another question to the forum asking how to raise this as a bug with Splunk (if that's even possible) but for now I'm going to have to code my deployment script around this problem by first putting sslPassword in a different section, restarting splunk to get the hashed value, and then moving it back to the global section (nasty).

Thanks for your help.

[cam343]

Also here is my outputs.conf I used on a recent deployment, perhaps a change in syntax may help?

[tcpout]
defaultGroup = default-auto-lb-group

[tcpout:default-auto-lb-group]
server = indexer01:9997, indexer02:9997
compressed = true

[tcpout-server://indexer01:9997]
sslCertPath = $SPLUNK_HOME\etc\auth\server.pem
sslPassword = password
sslRootCAPath = $SPLUNK_HOME\etc\auth\ca.pem

[tcpout-server://indexer02:9997]
sslCertPath = $SPLUNK_HOME\etc\auth\server.pem
sslPassword = password
sslRootCAPath = $SPLUNK_HOME\etc\auth\ca.pem

[cam343]

Hmmm according to: http://answers.splunk.com/answers/33385/splunk-hashes-the-sslpassword-making-it-so-that-the-key-file... Your sslPassword should be hashed if it's not in an "app".

Random question does: $SPLUNK_HOME/etc/auth/splunk.secret exist? This is the seed for the hashing.

Other thing to try would be chmod 777 temporarily outputs.conf and see if it's a write permission problem.