topic Re: /local/inputs.conf Not Being Read in All Apps and Add-ons

/local/inputs.conf Not Being Read

DBattisto — Mon, 18 Mar 2019 12:52:41 GMT

Hello all!

I'm experiencing an issue in my initial roll-out of my Splunk Universal Forwarder. While I had no issues in my test environment, I am now seeing an issue regarding /local/inputs.conf.

When I run btool ($splunkhome/bin/splunk btool inputs list --debug) it is not seeing my local/inputs.conf file. It appears that everything is the same as my lab environment, including permissions.

Does anyone have any suggestions as to what could be the cause of this issue? Thank you!

Re: /local/inputs.conf Not Being Read

nickhills — Mon, 18 Mar 2019 13:03:38 GMT

If you have the same stanza defined in multiple copies of the same file, there is an order of precedence which is followed.

Could it be that you have renamed your apps between prod/test?

my_app_prod will take priority over my_app_test,
however my_app_TEST, will take priority over my_app_prod
https://docs.splunk.com/Documentation/Splunk/7.2.4/Admin/Wheretofindtheconfigurationfiles

Re: /local/inputs.conf Not Being Read

DBattisto — Mon, 18 Mar 2019 13:50:00 GMT

Hey! Thank you for the response.

I don't think that's my issue. It was basically a 1:1 copy from the test environment, minus the change of the index name.

When I do the btool output, nothing from my local/inputs.conf file appears, but all of the default/inputs.conf file does.

Re: /local/inputs.conf Not Being Read

nickhills — Mon, 18 Mar 2019 14:36:49 GMT

Where are you running btool - on the server, or on the UF?
If it's on the server, are you expecting to see config from a deployment app?

Re: /local/inputs.conf Not Being Read

DBattisto — Mon, 18 Mar 2019 15:10:45 GMT

I'm running btool from the Universal Forwarder, which is on the endpoint device. I'm expecting to see the inputs file from the Windows_TA input that's loaded on the UF (as I do in my lab setting). So it's currently forwarding whatever was in defaults/inputs.conf and not local/inputs.conf

Re: /local/inputs.conf Not Being Read

nickhills — Mon, 18 Mar 2019 15:14:20 GMT

How did you configure local/inputs.conf for the forwarders?
Did you use a deployment server?

If so can you verify that $SPLUNK_HOME/opt/splunk/apps/deployment-apps/[YOUR UF APP]/local/inputs.conf exists and is properly defined for the clients?

Re: /local/inputs.conf Not Being Read

DBattisto — Mon, 18 Mar 2019 16:51:49 GMT

Yeah, the app deployed through my deployment server. The inputs.conf is located in deployment-apps/appname/local/ on the deployment server. I can also verify that the app has deployed to the UF's.

I'm going to try copy+pasting the inputs.conf to a different app on a UF to see if the problem is within the app (which is where this conversation is leading me to believe, thank you!).

Re: /local/inputs.conf Not Being Read

nickhills — Mon, 18 Mar 2019 16:59:17 GMT

I have had issues with the windows TA in the past.
We push a separate app which 'overrides' the Windows TA (this is where app naming becomes relevant for the precedence)
This means that the copy of the windows TA on the forwarders is an unmodified copy of the original package.

Re: /local/inputs.conf Not Being Read

DBattisto — Mon, 18 Mar 2019 18:19:53 GMT

Yikes. I don't know what to make of this. Basically renamed the '/default/inputs.conf' as '/default/inputs.conf.old' and restarted Splunk on the UF. I then ran btool and verified that my '/local/inputs.conf' file is now being acknowledged via:
`splunk btool inputs list --debug'

Re: /local/inputs.conf Not Being Read

DBattisto — Mon, 18 Mar 2019 18:22:40 GMT

I don't know what to make of this, but I solved it by renaming the '/default/inputs.conf' as '/default/inputs.conf.old' and restarted Splunk on the UF. I then ran btool and verified that my '/local/inputs.conf' file is now being acknowledged via:
splunk btool inputs list --debug

I then had to go back to the deployment server and make the change to send out to the clients. I know that modifying stuff in defaults is not recommended, and will get overwritten with the next app update. I have left a note for myself or a future admin to take note of this workaround until I can figure out what has actually happened.

Thank you to NickHillscpl for talking me through this and helping point me in the right direction.