topic Re: Splunk overwrites outputs.conf and inputs.conf on reboot in Getting Data In

Splunk overwrites outputs.conf and inputs.conf on reboot

wheatstraw — Fri, 11 Feb 2011 03:56:46 GMT

Greetings!

I'm trying to set up a splunk forwarder to use ssl between it and the indexer. I am only interested in encryption, so am trying to use the default ssl certificates. I followed the article, http://www.splunk.com/base/Documentation/4.1.2/Admin/EncryptandauthenticatedatawithSSL and when I restart splunk, it changes the files, removing a couple of options and changing the passwords to what looks like a random string.

Trying to outsmart it, I set outputs.conf on the forwarder and inputs.conf on the indexer with the immutable flag. (chattr +i outputs.conf) and I can see the error in logs where it tries to change the file. I was able to get it working like this, but don't want to have any 'immutable bombs' in my environment. 🙂

Has anyone else encountered this?

Thanks!

Pete

Re: Splunk overwrites outputs.conf and inputs.conf on reboot

David — Fri, 11 Feb 2011 05:02:36 GMT

What is the full path to the files you're modifying? Certainly at upgrade, any changes made to files in the "default" directories would be overridden, so maybe that's what's triggering it? Anything in "local" directories shouldn't be overwritten, though.

Re: Splunk overwrites outputs.conf and inputs.conf on reboot

bwooden — Fri, 11 Feb 2011 05:09:40 GMT

Splunk will update your local/inputs.conf and local/outputs.conf to obfuscate passwords for certificate files.

Splunk should not remove any options though. Would you be able to share a before and after of those two configuration files?

Re: Splunk overwrites outputs.conf and inputs.conf on reboot

wheatstraw — Mon, 28 Sep 2020 09:24:39 GMT

Before restart:

on the forwarder:

$ sudo cat /opt/splunk/etc/system/local/outputs.conf
[tcpout]
defaultGroup = ssl_group.domain.com_9997

[tcpout:ssl_group.domain.com_9997]
server = tico:9997

[tcpout-server://tico:9997]
sslCertPath=$SPLUNK_HOME/etc/auth/server.pem
sslPassword=password
sslRootCAPath=$SPLUNK_HOME/etc/auth/ca.pem
sslVerifyServerCert=false

Re: Splunk overwrites outputs.conf and inputs.conf on reboot

wheatstraw — Sat, 12 Feb 2011 00:07:05 GMT

ok, that was ugly. Going to try to submit my questions as an answer.....

Re: Splunk overwrites outputs.conf and inputs.conf on reboot

wheatstraw — Sat, 12 Feb 2011 00:07:50 GMT

Before restart:

on the forwarder:

$ sudo cat /opt/splunk/etc/system/local/outputs.conf
[tcpout]
defaultGroup = ssl_group.domain.com_9997

[tcpout:ssl_group.domain.com_9997]
server = tico:9997

[tcpout-server://tico:9997]
sslCertPath=$SPLUNK_HOME/etc/auth/server.pem
sslPassword=password
sslRootCAPath=$SPLUNK_HOME/etc/auth/ca.pem
sslVerifyServerCert=false

and the indexer:

$ sudo cat /opt/splunk/etc/system/local/inputs.conf
[SSL]
serverCert=$SPLUNK_HOME/etc/auth/server.pem
password=password
rootCA=$SPLUNK_HOME/etc/auth/cacert.pem
requireClientCert=false

[splunktcp-ssl:9997]

and after the restart of both forwarder and indexer on the forwarder:

$ sudo cat /opt/splunk/etc/system/local/outputs.conf
[tcpout]
defaultGroup = ssl_group.domain.com_9997

[tcpout:ssl_group.domain.com_9997]
server = tico:9997

[tcpout-server://tico:9996]
sslPassword = $1$gfv0UiKu6wR2
[tcpout-server://tico:9997]
sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
sslPassword = $1$gfv0UiKu6wR2
sslRootCAPath = $SPLUNK_HOME/etc/auth/ca.pem
sslVerifyServerCert = false

and on the forwarder:

$ sudo cat /opt/splunk/etc/system/local/inputs.conf

[splunktcp-ssl:9997]

[SSL]
password = $1$nZo1AKEa4vCH
requireClientCert = false
rootCA = $SPLUNK_HOME/etc/auth/cacert.pem
serverCert = $SPLUNK_HOME/etc/auth/server.pem

And when testing a quick "logger" command, I see nothing from the forwarder showing up on the indexer's search.
If I get nasty and do:

$ sudo chattr +i /opt/splunk/etc/system/local/outputs.conf
and
$ sudo chattr +i /opt/splunk/etc/system/local/inputs.conf
respectively on the forwarder and indexer, I can see splunk trying to overwrite my configs in the log:

02-11-2011 09:47:44.357 ERROR IniFile - Cannot rename /opt/splunk/etc/system/local/outputs.conf.tmp.30860 to /opt/splunk/etc/system/local/outputs.conf: Rename failed in 1 attempt(s) made between Fri Feb 11 09:47:44 2011 and Fri Feb 11 09:47:44 2011 [status code: 1] 02-11-2011 09:47:44.357 ERROR PropertiesMapConfig - Failed to save settings: /nobody/system/outputs/tcpout-server://tico:9996 (user: , app: , root: /opt/splunk/etc): Could not flush changes to disk: /nobody/system/outputs/tcpout-server://tico:9996/sslPassword: ConfPathMapper: /opt/splunk/etc/system/local 02-11-2011 09:47:44.357 INFO TcpOutputProc - ALL Connections will use SSL with sslCipher= 02-11-2011 09:47:44.357 INFO TcpOutputProc - initializing single connection with retry strategy for tico:9996 02-11-2011 09:47:44.381 INFO TcpOutputProc - attempting to connect to tico:9996... 02-11-2011 09:47:44.381 INFO TcpOutputProc - Will retry at max backoff sleep forever 02-11-2011 09:47:44.381 INFO TcpOutputProc - Using SSL for server tico:9997, sslCertPath=/opt/splunk/etc/auth/server.pem 02-11-2011 09:47:44.382 ERROR IniFile - Cannot rename /opt/splunk/etc/system/local/outputs.conf.tmp.30860 to /opt/splunk/etc/system/local/outputs.conf: Rename failed in 1 attempt(s) made between Fri Feb 11 09:47:44 2011 and Fri Feb 11 09:47:44 2011 [status code: 1] 02-11-2011 09:47:44.382 WARN PropertiesMapConfig - Error removing preexisting values: /nobody/system/outputs/tcpout-server://tico:9997 [ConfPathMapper: /opt/splunk/etc/sy stem/local] 02-11-2011 09:47:44.382 WARN TcpOutputProc - Failed to make a connection, will retry. 02-11-2011 09:47:44.384 ERROR IniFile - Cannot rename /opt/splunk/etc/system/local/outputs.conf.tmp.30860 to /opt/splunk/etc/system/local/outputs.conf: Rename failed in 1 attempt(s) made between Fri Feb 11 09:47:44 2011 and Fri Feb 11 09:47:44 2011 [status code: 1] 02-11-2011 09:47:44.384 ERROR PropertiesMapConfig - Failed to save settings: /nobody/system/outputs/tcpout-server://tico:9997 (user: , app: , root: /opt/splunk/etc): Could not flush changes to disk: /nobody/system/outputs/tcpout-server://tico:9997/sslVerifyServerCert: ConfPathMapper: /opt/splunk/etc/system/local 02-11-2011 09:47:44.384 INFO TcpOutputProc - ALL Connections will use SSL with sslCipher= 02-11-2011 09:47:44.384 INFO TcpOutputProc - initializing single connection with retry strategy for tico:9997 02-11-2011 09:47:44.384 INFO TcpOutputProc - attempting to connect to tico:9997... 02-11-2011 09:47:44.384 INFO loader - Instantiated plugin: controlqueueoutputprocessor 02-11-2011 09:47:44.384 INFO PipelineComponent - Pipeline merging disabled in *mode.conf file 02-11-2011 09:47:44.384 INFO PipelineComponent - Pipeline typing disabled in *mode.conf file 02-11-2011 09:47:44.384 INFO loader - Running.... 02-11-2011 09:47:44.384 INFO PipelineComponent - Launching the pipelines. 02-11-2011 09:47:44.384 ERROR IndexProcessor - received event for unconfigured/disabled index='_audit' with source='source::audittrail' host='host::localhost' sourcetype=' sourcetype::audittrail' 02-11-2011 09:47:44.385 WARN pipeline - Exiting pipeline tail gracefully: got eExit from processor tail 02-11-2011 09:47:44.385 INFO loader - Server supporting SSL v2/v3 02-11-2011 09:47:44.385 INFO loader - Using cipher suite ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM

And now, with the immutable flag set, I have ssl traffic from forwarder to indexer and log entries show up as expected.

Summary:

If I immute (is that a verb? 🙂 ) the outputs and inputs files, splunk can't muck with them and all works. I just don't want to do this in our production indexer for fear of leaving a little time bomb.
I could swear it wasn't just the passwords when working with it a week ago, but I've slept a bit since then. :)
Thank you so much for offering assistance!

Pete

Re: Splunk overwrites outputs.conf and inputs.conf on reboot

gkanapathy — Sat, 26 Feb 2011 08:00:55 GMT

You can edit the original question and update it too.

Re: Splunk overwrites outputs.conf and inputs.conf on reboot

Charlie — Mon, 28 Sep 2020 10:02:44 GMT

Found this "Note that the server certificate pass phrase will be hashed and stored in $SPLUNK_HOME/etc/system/local/inputs.conf, overwriting the clear-text value of "password" if it was defined there. If "password" was defined in clear-text in an inputs.conf located in an app, it will not be hashed there and will still be present in clear text in that location. This doesn't matter too much in this case since the pass phrase for the default server certificate is well known. " here - http://www.splunk.com/wiki/Community:Splunk2Splunk_SSL_DefaultCerts

Re: Splunk overwrites outputs.conf and inputs.conf on reboot

cmahan — Tue, 15 Sep 2015 16:37:30 GMT

Im having same issue on a Windows 2008 setup