Deployment Server: Support Dev & Prod enviroments without duplicating inputs.conf

[krussell101]

I have a single Indexer serving everything: production and test.

I've defined two servers classes: production and test. (I'm simplifying my world to better highlight my questions.)

I have two applications running in both test and production called "gateway" and "messagebroker". In prod they are running on their own servers. In test they share a server.

What I want to be able to do but don't know how:

QUESTION 1) Have a single outputs.conf file. I only have one indexer so I don't want multiple outputs.conf files. Where do I put this one outputs.conf file?

QUESTION 2) For production hosts I want the default index to be "prod" with some sources going to "prodsyst", likewise "test" and "testsyst" for test hosts. So I'd like an inputs.conf file for production hosts that looks like this:

[global]
index = test

[monitor:///var/log]
index = testsyst

In the specific app inputs.conf I will not define indexes. Is this possible and if so where do I put these two inputs.conf files?

QUESTION 3) For every instance of the "messagebroker" app I want to add specific directories to monitor that will be same for both prod and test, thus only a single inputs.conf file for messagebroker. In short, I don't want a serverclass file that includes [serverClass:productionservers:app:messagebroker] and [serverClass:testservers:app:messagebroker].

I really appreciate your help. I have read the splunk documentation and understand their examples, but their examples don't seem to cover this. Furthermore, it's the not the creation of the serverclass.conf file that puzzles me. It's what directories to create on the deployment server to hold my various config files. THANKS!

Here is my serverclass.conf on my deployment server:

---

[global]
whitelist.0=*

[serverClass:linuxservers]
whitelist.0 = *.*.mycompany.inc

[serverClass:productionservers]
filterType = whitelist
whitelist.0 = *.prod.mycompany.inc

[serverClass:testservers]
filterType = whitelist
whitelist.0 = *.test.mycompany.inc

[serverClass:linuxservers:app:gateway]
stateOnClient=enabled
restartSplunkd=true

[serverClass:linuxservers:app:messagebroker]
stateOnClient=enabled
restartSplunkd=true

---

[dwaddle · Accepted solution]

Question #1 -- my advice would be an app specific to outputs.conf. As your Splunk deployment evolves, you may have many other servers running dozens of applications all running forwarders. The commonality of all of these is they need to be able to tie into your indexers. So, it makes sense to build a mydotcom-forwarderconfig app that deploys outputs.conf, limits.conf, and whatever else you need that is "standard" forwarder config.

[krussell101]

Cool. Think I have it now. Thanks for the all the help!

[sowings]

Yes, they are additive. To avoid unexpected collisions, you may want to place your indexes.conf in your thisismyindexapp's local subdirectory, acting as though it were an override to the config. This means that thisismyindexapp's edits will take precedence over defaults, whether system wide or application-level. Note that configs in system/local are like trump cards--they override even items in application local directories.

[krussell101]

Got it.

I suspect I'm thinking of "app" too literally. So your suggestion could also be the answer to the rest of my questions.

I could have an serverclass:productionservers:app:thisismyindexapp which define the indexes for all production servers.

Configuration files are additive too, correct? I can have an inputs.conf in both productionservers:app:thisismyindexapp and linuxservers:app:messagebroker?

[dwaddle]

In your example, it would belong to the linuxservers class, or an everyone class that represents ALL of the forwarders. Remember, classes are additive - a host can be a member of many classes.

[krussell101]

Apps have to belong to serverclasses as I understand it. What serverclass would this app belong to?