Best practices for installing and configuring apps in a distributed Splunk install?

[Justin_Grant]

This is a question for Splunk users who have installed apps from Splunkbase into a distributed Splunk environment, meaning you have separate forwarders to collect data and search heads/indexers for indexing and searching.

What are the best practices you've figured out for how to download, install, configure, and distribute Splunk apps across your distributed install?

Do you use the Splunk GUI (the "install free" button in Launcher and Manager, as well as the setup pages which come up when you run some apps for the first time) to install and configure apps onto a test server (not a forwarder), and then once your app is configured the way you want it, manually deploy the config out to your forwarders?

Or do you generally download the package from Splunkbase, unpack it yourself, and manually configure it without using the GUI?

And how do you handle the requirement that app configs must be different across different server types in a distributed environment (e.g. apps on search heads shouldn't have inputs enabled)?

Also, how do you handle updating your customized configs once new versions of an app come out?

Finally, any other advice or suggestions for using Splunk Apps in a distributed environment?

Full disclosure: I'm asking these questions because we're trying to figure out to make Splunk easier to deploy apps in a distributed environment. So I'm less asking for my own benefit and more so I can understand how folks are solving these problems today so we can make it easier for you.

[DrewO]

Follow up question: This splitting into different names is partly for sanity and also partly to make Deployment Server happy non? If it weren't for Deployment server I don't think we'd call bundles of config files apps.

[gkanapathy]

oh yeah, and i don't care that they're called apps. they're just bundles of config files for this entire thing. except I suppose the one that i just call myapp, which might be considered "the" app.

[gkanapathy]

Well, no, it's applicable for any distributed environment. I do the same thing if puppet (for example) is used. It is more because the separate roles of a Splunk server (input, parsing, indexing, search, etc. see http://www.splunk.com/wiki/Where_do_I_configure_my_Splunk_settings%3F ) that handle each separate phase of the data life cycle can potentially be on different machines. Even on a single server install, it is best to do this so that some day in the future you can add or rearrange servers easily, e.g., switch from light to heavy forwarders, add a search head, etc

[gkanapathy · Accepted solution]

Here's how I do split up an "app". Basically, I break it up into multiple Splunk apps, so that the right pieces can be sent to the right place. For most apps, I'd have:

• myapp_inputs: This would contain the input configurations and is sent to forwarders. Mostly it's in inputs.conf, but would include scripted inputs and the input-time props.conf settings. (See this for what is input-time configuration.)
• myapp_props: This basically contains index-time and search-time data properties. I usually send it to all servers (forwarders, indexers, search heads) because it doesn't hurt to have the extra configurations, and it saves having to worry about whether a particular configuration is search-time or input-time or index-time. It also lets you forget about whether a forwarder is light or heavy. Basically, this is stuff in props.conf and things referenced from there. (e.g., transforms.conf, lookup tables and scripts, search scripts)
• myapp_summarize: Goes on the search head that runs summary jobs. This has the summary searches that generate summary index data.
• myapp: This contains dashboards and views, and just goes to the search head. It will also include most saved searches, and usually includes alerts. Sometimes i'll put the views in the myapp_props and myapp_summarize apps instead. I'll include some of the search-time configurations here, if those configurations are more purely view-oriented rather than data-oriented (e.g., lookup tables used to populate form dropdowns, vs lookup tables used to enrich events).

I suppose there could be app-specific configs for something like data routing, but I haven't seen that configuration at application-level (vs being managed at the base server server level). Alerts and (non-summary) scheduled searches may be split from the myapp app into a myapp_jobs app, but that's only of benefit if you're getting to the point where you have scaled to a separate search head for users vs jobs.

---

This is recommended (by me) even if you have a non-distributed Splunk install, as it:

• gives the flexibility to add and reconfigure your system, e.g., add a search head, split summarization away, change forwarders from heavy to light, without having to change and split up config files
• corresponds roughly to different deployment phases, and different tasks/roles/people who will be involved at each phase; e.g., dashboards and views are usually handled in a different role or by different people from the data inputs.

---

Lately, I have been also splitting up the props into two apps, myapp_props and myapp_indexprops to distinguish search-time from index/parse-time configurations. I am not as certain that this is that important to separate however.

[Lowell]

Gkanapathy, any updated recommendations for Universal Forwarders? Specifically, for the myapp_props, is there any reason to deploy props to a uf?

[tpsplunk]

you also have to worry about whether the app defines a new index or uses an existing one. this thread covers a lot of the salient points.: http://splunk-base.splunk.com/answers/10980/app-development-best-practice-new-index-or-not