All Apps and Add-ons
Highlighted

What are the best practices for S.o.S - Splunk on Splunk to monitor a Search Head Cluster tied with a Multisite Indexer Cluster?

Path Finder

I need to install S.o.S - Splunk on Splunk to monitor a Multisite Indexer Cluster and Searh Head Cluster:
2 Sites with 2 Peers Each
3 SH on the SH Cluster
Master and Deployer reside on the same VM.
Multiple Forwarders.

The documentation that's available on Splunk Docs for Planning the SOS installation does not cover SH Cluster.

0 Karma
Highlighted

Re: What are the best practices for S.o.S - Splunk on Splunk to monitor a Search Head Cluster tied with a Multisite Indexer Cluster?

Builder
Highlighted

Re: What are the best practices for S.o.S - Splunk on Splunk to monitor a Search Head Cluster tied with a Multisite Indexer Cluster?

Splunk Employee
Splunk Employee

In addition to the Splunk Answer that @sk314 points out, I will indicate that as of right now S.o.S has not been certified to work without a hitch on a search-head cluster.

I would expect that the app will work fine, with one exception: It's dynamically generated lookups which are used as asset tables and are maintained on a per-instance basis which may not be entirely compatible with an SHC.

In order to circumvent that, you'll probably need to do something like this:

  • Push the latest version of Sideview Utils to your SHC via its App Deployer
  • Stage the S.o.S app before pushing it to your SHC via its App Deployer by renaming every *.xml.bak file to *.xml in the app's default/data/ui/views. This can be accomplished with the disable_FTR.sh script located in that directory.
  • Push the S.o.S app to your SHC via its App Deployer
  • Disable the "sosrefreshsplunkserverscache" scheduled search in the S.o.S app
  • Populate the $SPLUNK_HOME/etc/apps/sos/lookups/splunk_servers_cache.csv table manually instead, following the specifications recorded in splunk_servers_cache.spec.
Highlighted

Re: What are the best practices for S.o.S - Splunk on Splunk to monitor a Search Head Cluster tied with a Multisite Indexer Cluster?

Champion

I would advise against use sos be instead configure Distributed Management Console. The DMC has a lot more information and is more responsive. Works well with clustered search heads and indexers. It really easy to setup

Highlighted

Re: What are the best practices for S.o.S - Splunk on Splunk to monitor a Search Head Cluster tied with a Multisite Indexer Cluster?

Splunk Employee
Splunk Employee

A fine point indeed, @bmacias84! If you are running Splunk 6.1 or above, you should be setting up a Distributed Management Console to monitor your Splunk deployment.

Highlighted

Re: What are the best practices for S.o.S - Splunk on Splunk to monitor a Search Head Cluster tied with a Multisite Indexer Cluster?

Builder

I recently installed S.o.S on our clustered Splunk(6.2) instance. As per @bmacias84, I went ahead and configured DMC. It's really cool! Thanks for the tip.