All Apps and Add-ons

What are the best practices for installing SoS on cluster?

Path Finder

As written above - I just set up a cluster (Master, 2 Indexers + SearchHead). Are there some good practices for installing SoS on cluster? Is it also pushed from teh master to the nodes or installed on search head as in distributed deployment?

1 Solution

Splunk Employee
Splunk Employee

@alacercogitatus has the gist of it but there are a few more details.

This is a 3-step process:

NB:

  • Don't forget to enable the scripted inputs ( lsof_sos.sh and ps_sos.sh) before pushing them out.
  • Due to a Splunk bug (SPL-64308), the file permissions of scripted inputs distributed by the cluster configuration bundle on the peers will be incorrect and will need to be manually corrected. This will be fixed in maintenance release 5.0.4.

View solution in original post

Path Finder

If the Search Heads are not forwarding sos logs to the indexing cluster you should create a static distributed peer for each SH on the master.
It´ll then see Search Heads logs althought they will be listed as Indexers on the topology.

Another way is to forward sos logs from the Search Head just to the Master if this could not be added to the indexing cluster disk usage.

0 Karma

Splunk Employee
Splunk Employee

@alacercogitatus has the gist of it but there are a few more details.

This is a 3-step process:

NB:

  • Don't forget to enable the scripted inputs ( lsof_sos.sh and ps_sos.sh) before pushing them out.
  • Due to a Splunk bug (SPL-64308), the file permissions of scripted inputs distributed by the cluster configuration bundle on the peers will be incorrect and will need to be manually corrected. This will be fixed in maintenance release 5.0.4.

View solution in original post

Path Finder

Is it any different if the Search Heads are part of a SH Cluster? Still follow the same steps?

0 Karma

Builder

I have a similar setup (2 search heads in a cluster, 3 clustered indexers).It's the same process. Install the S.o.S app on the search heads using the deployer.

0 Karma

Splunk Employee
Splunk Employee

Or better yet: Since you are running Splunk 6.2, use the Distributed Management Console to monitor your Splunk deployment.

Note that as of right now, S.o.S is not certified on search-head clusters. Read more this here.

0 Karma

Contributor

"Install the S.o.S app on the cluster master. This is the only instance where the "Cluster Master View" will work."
Is it the case for situation when I forward all indexes from Cluster Master to Indexers layer?

0 Karma

Splunk Employee
Splunk Employee

Yes, that will work fine.

0 Karma

Contributor

Is it really the ONLY instance where the "Cluster Master View" will work" in case of forwarding of indexes?

0 Karma

Splunk Employee
Splunk Employee

Yes, because that view (and the "Bucket Fix-up Activity" view) depend on searches against the local (in other words, the Cluster Master's) REST API to be populated.

0 Karma

Contributor

Ok, thank you for clarifying!

0 Karma

Splunk Employee
Splunk Employee
  1. You're welcome.
  2. You need to manually change the permissions on the peers etc/slave-apps/$APP/bin/* files.
  3. Not at this time. It will definitely be fixed with our next maintenance release (5.0.4) and may be a part of a special patch before that.
0 Karma

Communicator
  1. Thanks for the great response.
  2. The permissions in SPL-64308 can be change on the slave-apps directory on the peers or does it have to be changed in the bundle and redeploed?
  3. Is SPL-64308 available to the public?
0 Karma

Path Finder

Thanks a lot for the detailes!

0 Karma

SplunkTrust
SplunkTrust

You may want to consult this doc:

http://docs.splunk.com/Documentation/Splunk/5.0.2/Indexer/Clustersinscaledoutdeployments#Cluster_pee...

To Install SoS in a cluster, install the app on the search head, and the TA-sos on the indexers. Or use deploy-server to send the TA to the indexers.