Create indexer app purely for index configs and splitting hot\warm paths - what considerations when ES Splunk_TA_ForIndexers is used ?

[Skins]

allo,

I have inherited a scenario of 1 x SH, 1 DS, 1 IDX, 1HF

The SH has an instance of ES installed. I'm looking at moving the hot\warm buckets to the SSD drive and leaving the cold\frozen in the old location.

the indexer.confs seem spread out across the deployment and found most of them defined in the ../etc/apps/search/local/indexes.conf

To try and tidy this up a bit would you recommend creating a new app on the DS and copying the /etc/apps/search/local/indexes.conf to the new app (index_master), making my changes to point the hot\warm to the new location and deploying it to the IDX creating an index master config ?

** also aware of manually copying the files across **

Are there any other possible conflicting considerations with the Splunk_TA_ForIndexers app which is also installed on the IDX and has a series of indexes defined ?

have seen the article here also https://answers.splunk.com/answers/368098/splunk-enterprise-security-requires-a-deployment-c.html

gratzi

[woodcock]

It is a best practice not to use Splunk_TA_ForIndexers; it is a brute-force, brainless superset of everything that goes anywhere and 95% or more of it has nothing to do with the Indexers.

This just came up in #enterprise_security; here is my paraphrase of that conversation (many thanks to @beatus and @automine)...

The packager for Splunk_TA_ForIndexers isn't smart; it just says "Gimme all the indexes.conf, props.conf and transforms.conf for ES" and sticks everything in an app.
There is no "harm" in having all that junk on the indexers, but most of that stuff is totally un-necessary; it is a brainless "easy button" so is not precise. As a rule of thumb, other than indexes.conf, anything from SA-* or DA-* doesn't need to exist on the indexers, at least for those that ship with ES. The docs give you a handful of ways to accomplish the same thing:
http://docs.splunk.com/Documentation/ES/4.7.2/Install/InstallTechnologyAdd-ons#Deploy_add-ons_to_ind...
See the table there? It says pretty much: "we made this thing for you. You can use it or do it yourself".
Keep the rule in mind that only the "first full instance of Splunk" needs the props/transforms, because that's the guy that cooks the data.
So what everyone should do when deploying any so-called "TA" is think about the path of the data as well as see what is actually in the app (what it really does).
Take DBX, for example; his inputs start at the Splunk server with DBX installed. I know that DBX is installed on a full instance of Splunk, therefore the TA should be there. However file monitors are probably done by a UF, which probably sends to the IDX. Therefore the IDX needs the TA for those.
https://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F
http://docs.splunk.com/Documentation/Splunk/6.6.3/Indexer/Indextimeversussearchtime
http://docs.splunk.com/Documentation/Splunk/6.6.3/Deploy/Datapipeline
http://wiki.splunk.com/Community:HowIndexingWorks

@beatus++

[jkat54]

Typically on PS engagements we create what is known as "base apps".

Something like {org_name}_indexes_base will typically fit the bill.

Then depending on the architecture, this gets deployed via the cluster master (to clustered peers), or deployment server (to standalone search heads), or deployer (to search head clusters).

So you're heading in the right direction.

As for conflicts, remember that splunk will "merge" the configurations together in ASCII order of precedence, giving configs in /local/indexes.conf priority over those in /default/indexes.conf. So as to say appName_AAA/default/indexes.conf will override the same stanza keys in appName_ZZZ/default/indexes.conf when they get merged together.

An example is below:

appName_AAA/default/indexes.conf
[indexName1]
key1 = value111

appName_ZZZ/default/indexes.conf
[indexName1]
key1 = value222
key2 = value333

Merged together would become:
[indexName1]
key1 = value111
key2 = value333

Its easier to see how they merge together once you put the configs in place and use the following command:

splunk btool indexes list --debug

You can even put the configs in place, run the command and if all looks well, then restart splunk... if you prefer. Of course you cant do that when deploying from a deployment server, etc. using restartSplunkD=true in the serverclass.