I run Splunk 6.1.4 in a distributed environment:
In short: there's surprisingly little documentation for the SA-Hydra and SA-Utils apps (even the READMEs aren't very telling)- they both seem really heavy and I'm ambivalent about having them broadly deployed without a more thorough understanding... in particular, other than the event parsing in props.conf I don't see anything in either app that specifically is needed on the indexers (true?)...
There looks to be a lot happening in the SA-utils app and I don't have a warm and fuzzy on what's really going on. Consider:
[script://./bin/tsidx_clean_up.py] disabled = false passAuth = splunk-system-user # Once per day at 3 AM interval = * 3 * * * index = _internal sourcetype = tsidx:clean_up
Isn't Splunk doing this on its own already? I'm not clear why this is needed...
There's a lot going on under the covers here and without understanding this a little better I'm a little uncomfortable deploying it. On the other hand the Splunk App for VMWare is one of the top requests from my infrastructure folks so I want to support them. Can anyone shed some light on these apps?
TL;DR: I'm a bit of a minimalist when it comes to the configs at each stage of the pipeline. I don't like to have irrelevant configs where they have no business (for instance, an indexes.conf on a forwarder, inputs.conf on the indexers... don't even get me started on props and transforms). To that end, when I'm looking at a new app I dig through and dissect apps so that only relevant configs are present. I know... Splunk is good at ignoring irrelevant configs if they don't apply, and having these bundled apps is easier on the developers with all of the various Splunk architectures, but I find it is a lot easier, for me, to look at and resolve conflicts on an ongoing basis if I minimize the configs up front.
So when I look at the Splunk App for VMware component reference and where the various components need to be installed, it's making my head hurt. There's a lot going on in there and I just really need to understand more deeply what all of these pieces are really doing in support of the VMWare data.
Technically they do not have to go on the Indexer to make the app function. I have been told that SA-Utils and SA-Hydra are recommended only because they will stop modular input introspection from failing.
I had a call with the developers and after we chatted for awhile it became apparent that the the posted documentation was somewhat inaccurate- it's since been updated. SA-Hydra and SA-Utils aren't to be installed on dedicated indexers (see the updated Splunk App for VMware component reference)
the updated documents still show SA-Hydra and SA-Utils being installed on the indexers, both in the specific version you linked (3.1.2) as well as the latest version (3.1.3, at the time of this writing).
hmph... yeah- looks like it was revised as stated on 14 Nov, but modified again on 21 Nov to the prior state of listing SA-Hydra as an indexer component, according to the article history. Looks like the note on the Introspection workaround got added in Mar 2015.