Distributed Deployment: Indexers - Apps or No-Apps?

[nickhills]

I am preparing for a new fresh deployment which is expected to take a lot of data (and users) going forwards. For this reason i want to get the distributed deployment right from day 1 to make it easier to scale out in the future.

To this end, I have the following:
1 deployment server (with exported folder for Search Head pooling)
2 Search Heads (with shared storage for pooling)
2 Indexers.
no forwarders - yet

The Search heads are not managed by the deployment server (as i can make changes to the nfs export) but my indexers are managed by the deployment server.

I have a DM app called ‘indexers’ which configures (among other things), LDAP auth, and some custom indexes we use.

This all works beautifully, and has been quite painless so far, but i have hit my first problem.

Having installed both the *nix and Deployment Manager apps on the search heads i am getting warnings about missing indexes on the indexing servers. This of course makes sense, as those indexes are created when the applications are installed, and no such installation has been performed on my (nice clean) indexers.

My question therefore is this:

For applications which include custom indexes, is it recommended to install the app on the indexer, or simply create the index by another means (ie deployment server)?

I can see pros and cons for both:
-Keeping apps up to date on a large number of indexers could be a pita
but
Custom index deployment could be prone to mistakes

What is the general consensus on this?

[nickhills · Accepted solution]

This question was made moot by the introduction of clustering in Splunk5.

Now in my clustered environment, the Master handles my index management, and I don't have to worry about pushing indexes.conf anymore!

Thanks to the contributors above, and thanks to Splunk!

[tskinnerivsec]

In our distributed deployment, I use a deployment app who's only task is to create indexes is and I have it assigned to just my indexer class. This makes it really simple to create custom indexes if I need them for splunkbase apps or any other app I would create to collect new data. I use a lot of custom indexes because we have many different classes of users who shouldn't see all of the log data, just what is pertanent to them. So, using custom indexes for my various data sources allows me to restrict users' views with ACLs.

[MHibbin]

I'd say install the App, instead of creating the custom indexes, at least then there is a uniform standard. There will also be correct replication of any other configurations that may be required by that App.

Among other things this will help narrow down any troubleshooting as there will be a standard, and as such should make support cases easier (internal or external.