What are the best practices for installing SoS on cluster?

[jtworzydlo]

As written above - I just set up a cluster (Master, 2 Indexers + SearchHead). Are there some good practices for installing SoS on cluster? Is it also pushed from teh master to the nodes or installed on search head as in distributed deployment?

[theunf]

If the Search Heads are not forwarding sos logs to the indexing cluster you should create a static distributed peer for each SH on the master.
It´ll then see Search Heads logs althought they will be listed as Indexers on the topology.

Another way is to forward sos logs from the Search Head just to the Master if this could not be added to the indexing cluster disk usage.

[hexx · Accepted solution]

@alacercogitatus has the gist of it but there are a few more details.

This is a 3-step process:

• Install the S.o.S app on the search-head(s).
• Install the S.o.S app on the cluster master. This is the only instance where the "Cluster Master View" will work.
• Install the S.o.S technology add-on on the peers by means of the cluster configuration bundle. Deployment server is not a supported method to push content to cluster peers.

NB:

• Don't forget to enable the scripted inputs ( lsof_sos.sh and ps_sos.sh) before pushing them out.
• Due to a Splunk bug (SPL-64308), the file permissions of scripted inputs distributed by the cluster configuration bundle on the peers will be incorrect and will need to be manually corrected. This will be fixed in maintenance release 5.0.4.

[singhbc]

Is it any different if the Search Heads are part of a SH Cluster? Still follow the same steps?

[sk314]

I have a similar setup (2 search heads in a cluster, 3 clustered indexers).It's the same process. Install the S.o.S app on the search heads using the deployer.

[hexx]

Or better yet: Since you are running Splunk 6.2, use the Distributed Management Console to monitor your Splunk deployment.

Note that as of right now, S.o.S is not certified on search-head clusters. Read more this here.

[andrey2007]

"Install the S.o.S app on the cluster master. This is the only instance where the "Cluster Master View" will work."
Is it the case for situation when I forward all indexes from Cluster Master to Indexers layer?

[hexx]

Yes, that will work fine.

[andrey2007]

Is it really the ONLY instance where the "Cluster Master View" will work" in case of forwarding of indexes?

[hexx]

Yes, because that view (and the "Bucket Fix-up Activity" view) depend on searches against the local (in other words, the Cluster Master's) REST API to be populated.

[andrey2007]

Ok, thank you for clarifying!

[hexx]

1. You're welcome.
2. You need to manually change the permissions on the peers etc/slave-apps/$APP/bin/* files.
3. Not at this time. It will definitely be fixed with our next maintenance release (5.0.4) and may be a part of a special patch before that.

[agodoy]

1. Thanks for the great response.
2. The permissions in SPL-64308 can be change on the slave-apps directory on the peers or does it have to be changed in the bundle and redeploed?
3. Is SPL-64308 available to the public?

[jtworzydlo]

Thanks a lot for the detailes!

[alacercogitatus]

You may want to consult this doc:

http://docs.splunk.com/Documentation/Splunk/5.0.2/Indexer/Clustersinscaledoutdeployments#Cluster_pee...

To Install SoS in a cluster, install the app on the search head, and the TA-sos on the indexers. Or use deploy-server to send the TA to the indexers.