How to manage global outputs.conf for LWF's

[chicodeme]

I am rolling out splunk to thousands of linux/sunos,aix based servers.

I have deployment server setup to distribute via machinetype. So for the different machinetypes i will be monitoring different inputs: AIX: /var/adm/syslog/syslog.log SUN: /var/adm/messages Linux: /var/log/messages

The deployment manager works all fine and dandy.

On the client side rollout, I am rolling it out with the LWF mode ./splunk enable app SplunkLightForwarder I decided to issue the following in my client side rollout script ./splunk add forward-server servername:9997 ./splunk disable local-index After issuing those commands the LWF works as expected. All is good. The problem is that the add forward-server and disable local-index command line stored the config in: /opt/splunk/etc/apps/search/local/outputs.conf While it works, I would prefer it be stored as a global setting here: /opt/splunk/etc/system/local/outputs.conf I could not see a parameter to pass to do this via the cmdline. Of course I could just echo the output to the exact file I wanted, but I like to use the provided utils in case any of the text w/n outputs.conf changes/deprecates/varies/etc.. It seems odd that the cmdline would store the results in the search app... but that is exactly the app your in at the cmdline so it makes sense. http://www.splunk.com/base/Documentation/4.1.3/Admin/Configureforwarderswithoutputs.conf "When you enable a forwarder through Splunk Web or the CLI, Splunk creates an outputs.conf file in the directory of the currently running app. For example, if you're working in the search app, Splunk places the file in $SPLUNK_HOME/etc/apps/search/local/."

This then got me thinking how I would want to manage these config settings I am changing. It would be nice to manage them via the Deployment Server and have them global. This doesn't seem possible as everything I have tried/read/thought of would leave it at the app level. For example, I could take the LWF and set it up to be a deployed app via the deployment server with all the configs I need.. But then I have my outputs.conf settings set at the app level. That is fine as well if I want to name it like 1lwf_app... (that is the number 1 and then an l short for light)... per http://www.splunk.com/base/Documentation/latest/Admin/Wheretofindtheconfigurationfiles section: "How app directory names affect precedence"

I want my fowarding and indexing settings at the global level, so I don't have to worry about the app names and it just makes organizational sense to me. Then, I would like the deployment server to manage those.

This link http://www.splunk.com/base/Documentation/4.1.3/Admin/Configureforwarderswithoutputs.conf recommends just having one outputs.conf in via the deployment server, but again now i have to worry about app naming global/app & app naming precedence. "No matter where the outputs.conf file resides, it acts globally on the forwarder (bearing in mind the issue of location precedence, as described in Configuration file precedence). For purposes of distribution and management simplicity, you might prefer to maintain just a single outputs.conf file, keeping it resident in an app."

You can't have the deployment server manage the global configs files that i could find: /opt/splunk/etc/system/local/outputs.conf

Will the future of splunk allow me to manage global configs? If so, will it be as a special "appname"? If its not on the radar, then I will just leave it at the app level and manage it via an app based on how deployment server currently works or echo out my file directly via my script. Heck I might even leave it in the search app config just so the LWF is deployed and working properly client side. For all I know if I set it up and the LWF complains about no outputs.conf existing it might just keep waiting for that before it goes and polls the deployment server to download the outputs.conf it needs. And if that is the case, then well that option is not an option. Anyone want to cast their vote?

I am rather new to splunk so please point out any flaws or other solutions that you know.

[gkanapathy · Accepted solution]

I think the simplest solution, and the one everyone who uses Deployment Server uses, is to set the configuration in an app folder, and not in system. Where the config is stored is purely a matter of convenience in managing it. It is easiest if it is in (for example) etc/apps/search/local or etc/apps/SplunkLightForwarder/local or etc/apps/myforwarderconfigs/local/, then just put it there.

[chicodeme]

Your comment "Where the config is stored is purely a matter of convenience in managing it." is a little short sighted based on the above detail I mentioned. Where config files are stored represent far more than convenience. They matter for organizational reasons, inheritance and precedence purposes as well.

Nonetheless, I can see that no one really cares about this and everyone seems to lean towards just the management aspect of things, which means that you should just manage it as an app in Deployment Server.

Thanks for taking the time to offer your opinion and insight.