etc\apps\SplunkUniversalForwarder\local\inputs.conf overwriting other apps

[Choi_Hyun]

Hello, everyone.

I just ran into an issue where a stanza within apps\SplunkUniversalForwarder\local\inputs.conf on a forwarder is overwriting other apps\AppName\local\inputs.conf  from other apps in the apps folder.

I would like to either disable this app, or delete the \SplunkUniversalForwarder\local folder or delete the stanza.

The problem is that this has happened on multiple hosts and I need an automated method of doing this.

Does anyone have an idea so that this default app that I don't even want to touch doesn't overwrite my own actually used apps?

Thanks

[PickleRick]

Ok. That's interesting because the SplunkUniversalForwarder app is an app which indeed as @gcusello pointed out comes with your UF installation but it typically does not contain the local directory. As far as I remember the configurations you make with CLI splunk commands (like splunk add monitor) land in etc/system/local directory so they should not be there either.

While technically you can make changes to the default apps you shouldn't do so because in case of upgrade you'll overwrite the changes in apps that come with the installation package with your own versions again which might be undesirable. So you should not touch the default apps.

So I'd try to see where did those settings come from - either someone configured them manually (which is the "least bad" case here because on upgrade the "default" directory should get overwritten but "local" should should stay untouched) or your DS is serving this app (in which case you might want to check where it is being pushed to).

Anyway, if it's been done manually, you can always just do your favourite configuration automation software (ansible?) and just remove the file from your UFs. Or you can just deploy an app with a higher precedence which will override the settings from the problematic config. See https://docs.splunk.com/Documentation/Splunk/9.1.1/Admin/Wheretofindtheconfigurationfiles

[Choi_Hyun]

My DS doesn't have an explicit stanza to push SplunkUniversalForwarder app. I know that the file the other app the DS pushed did not exist on all hosts the app got pushed to.

Is it possible Splunk automatically decided to create a stanza on one of its input.conf file because it kept finding out that the file did not exist? We now have this file being logged on all hosts so now I have to manually change the input.conf file.

I also thought that my other non-default apps took precedence before the default SplunkUniversalForwarder app, but when I ran btool, it told me the file I was looking for was obtaining its configuration from \etc\apps\SplunkUniversalForwarder\local\input.conf instead of anywhere else.

What's truely strange is that this behavior is only happening on some hosts and not others.

[gcusello · Accepted solution]

Hi @Choi_Hyun,

the UniversalForwarder App is an internal Splunk App and usually it isn't used to add configurations, how do you have an inputs.conf in this App?

Anyway, I'm not sure that it's possible to manage this App using a Deployment Server, but if you have the inputs.conf file in local you could try to deploy this App with an inputs.conf with this stanza disabled.

Otherwise, the only solution is a remote script shell that remove this file (not the App!) and restarts Splunk.

I'm very confident about this last solution.

If all the above solutions don't work, open a case to Splunk Support.

Ciao.

Giuseppe

[Choi_Hyun]

Hi Giuseppe,

Thank you for your response.

I also have no idea why an input.conf file was created or how it was created.

I will test to see if my deployment server can push out an empty input.conf file to that folder, otherwise I might just have to use PowerShell to just delete and replace that file on our hosts.

To make it clear, this behavior is unusual right?

[gcusello]

i @Choi_Hyun ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

[gcusello]

Hi @Choi_Hyun,

as I said, for my knowledge, that app shouldn't be used for inputs.

also because it isn't possible to manage it by Deployment Server.

Ciao.

Giuseppe