What are best practices when deploying apps across an indexer clustering environment, and to which nodes?

[Cuyose]

We have a cluster master managing 2 indexers where we can push our indexes.conf in master apps, we have a single search head ( this is lab env), and multiple forwarders. When creating an app to deploy to the forwarders with inputs.conf, props.conf, and possibly transforms.conf, what serverclasses should I be pushing to?

I have read it's good practice to push the app out to the indexers and search head(s) as well as the actual universal forwarders. Is this the preferred method?

[scruse · Accepted solution]

So there's a couple of answers to this and it depends on your environment.

Are you doing search time extraction or index time extraction?

If search time you'll want your TA's on the search head server class, if index time, the indexer class.

The inputs apps will need to go in the class with your forwarders (be they heavy, lightweight, or universal)

Since you are doing clustering, you could make a serverclass that contains only the cluster master to push apps to your indexers by way of pushing them to the cm and then logging into the cm and applying the cluster bundle. But you don't really want to be managing your cluster through a DS as a matter of best practice. I would however use the DS in your situation for the SH, unless you also have SHC enabled.

In my environment, (very large, 500 indexers, 26 search heads) we use salt states to take care of all of this, but have individual states for varying aspects of the configuration. why im bringing this up is I would recommend structuring your serverclass similarly when writing it.

For instance, if you are using custom SSL certs for splunkweb, andthat needs to go to your CM, DS, SH, and LM, then you make an ssl serverclass that manages just that aspect.

Similar situation for pushing out your licensing app, etc.

Hope I explained this clear enough, if I didnt feel free to ping me in the usergroups slack @f8al. If you're interested in possibly going the saltstack route, I can provide you with a sanitized dummy config to look at to get an idea of what exactly we are doing for that segmentation.

[Cuyose]

Thanks,
It sounds like there are not many situations where you would want to do index time extractions, so you would use the deployment server to push a XXX_inputs app to only the forwarders on specified serverclasses, and push the XXX_props to both the universal forwarders and search heads?

We do have a cluster master where we manage our indexes, but light weight apps would only go to forwarders and search heads via deployment server I am assuming.

[scruse]

that is correct

[Cuyose]

There are docs indicating anything being deployed to indexers needs to go through the clustermaster, so I guess my question is for a simple "app" that monitors directories on an application machine with some path stanzas in inputs.conf, some sourcetypes configured in props.conf, and possibly a transforms.conf, what goes where and what pushes what?

[scruse]

indexers in a cluster are managed by the CM. you can do some clever hacks to have the slave apps dir be managed by a ds tho. its not considered best practice however.

The DS is utilized more extensively in a non clustered environment. without clustering you can use the ds for everything. Since you have clustering, your ds is now only for search heads and forwarders.