Is it harmful if all apps and add-ons are deployed to all servers (universal forwarders, search heads and indexers)? Will having extra inputs.conf or props.conf files cause issues during normal operations of any particular servers?

[mttilley65]

Do I really need to care if the props.conf or inputs.conf or apps get deployed to universal forwarders or search heads or indexers? If there are inputs.conf stanzas that are looking for file in a place that doesn't exist on a server will that cause performance issues?

[koshyk · Accepted solution]

I don't think you will see full documentation, but can say it is a bad practice to deploy all apps on all Splunk components

• Risk with inputs.conf => Same inputs.conf will cause data duplication and unwanted data coming. (eg. You had put to collect from /var/log/* on certain system. But this will be damaging on other systems as some application will put /var/log/myapplication/* and can fill your index within seconds)
• Risk with props.conf => Sourcetypes will be messed up if you don't properly align data. (eg. you monitor /var/log/syslog/app1 with sourcetype app1_sourcetype. But if there is another inputs with /var/log/syslog/*, you don't know how the sourcetype will be assigned)
• complexity with merged conf files=> if you run a btool , you will be surprised to see how each stanza would get preference. Basically, you will never expect to see things as per your code. Without naming convention and config file precedence control, you will be entering into wild wild west for enterprise systems.
• Risk of data going to wrong index => Of course, you don't have control where data goes as some default props/inputs will send to default indexes of the TA.
• Search Head Clusters => All the best !! With user inputs coming to local directory and your configs pushed to default, better to leave the company than to administer it 🙂

In my opinion, without best practices, control of configs and correct naming conventions it is very hard to administer in large enterprise systems.

[ddrillic]

Why would you do that? ; - )

[mttilley65]

It is not that I want to do this but the system that I am currently supporting was set up this way. I would like to change it but would like some documentation saying that this is not the best practice or that it may cause performance issues. I need to justify the time to change this.