SA-Hydra and SA-Utils in a distributed environment (subtitle: Splunk App for VMWare in a distributed Splunk environment... really)?

[jeff]

I run Splunk 6.1.4 in a distributed environment:

• dedicated search head
• dedicated indexers
• dedicated deployment server
• (mostly) universal forwarders

In short: there's surprisingly little documentation for the SA-Hydra and SA-Utils apps (even the READMEs aren't very telling)- they both seem really heavy and I'm ambivalent about having them broadly deployed without a more thorough understanding... in particular, other than the event parsing in props.conf I don't see anything in either app that specifically is needed on the indexers (true?)...

There looks to be a lot happening in the SA-utils app and I don't have a warm and fuzzy on what's really going on. Consider:

[script://./bin/tsidx_clean_up.py]
disabled = false
passAuth = splunk-system-user
# Once per day at 3 AM
interval = * 3 * * *
index = _internal
sourcetype = tsidx:clean_up

Isn't Splunk doing this on its own already? I'm not clear why this is needed...

There's a lot going on under the covers here and without understanding this a little better I'm a little uncomfortable deploying it. On the other hand the Splunk App for VMWare is one of the top requests from my infrastructure folks so I want to support them. Can anyone shed some light on these apps?

==========

TL;DR: I'm a bit of a minimalist when it comes to the configs at each stage of the pipeline. I don't like to have irrelevant configs where they have no business (for instance, an indexes.conf on a forwarder, inputs.conf on the indexers... don't even get me started on props and transforms). To that end, when I'm looking at a new app I dig through and dissect apps so that only relevant configs are present. I know... Splunk is good at ignoring irrelevant configs if they don't apply, and having these bundled apps is easier on the developers with all of the various Splunk architectures, but I find it is a lot easier, for me, to look at and resolve conflicts on an ongoing basis if I minimize the configs up front.

So when I look at the Splunk App for VMware component reference and where the various components need to be installed, it's making my head hurt. There's a lot going on in there and I just really need to understand more deeply what all of these pieces are really doing in support of the VMWare data.

[jeff · Accepted solution]

I had a call with the developers and after we chatted for awhile it became apparent that the the posted documentation was somewhat inaccurate- it's since been updated. SA-Hydra and SA-Utils aren't to be installed on dedicated indexers (see the updated Splunk App for VMware component reference)

[steven_swor]

the updated documents still show SA-Hydra and SA-Utils being installed on the indexers, both in the specific version you linked (3.1.2) as well as the latest version (3.1.3, at the time of this writing).

[jeff]

hmph... yeah- looks like it was revised as stated on 14 Nov, but modified again on 21 Nov to the prior state of listing SA-Hydra as an indexer component, according to the article history. Looks like the note on the Introspection workaround got added in Mar 2015.

[mgildenhorn_spl]

Technically they do not have to go on the Indexer to make the app function. I have been told that SA-Utils and SA-Hydra are recommended only because they will stop modular input introspection from failing.