All Apps and Add-ons
Highlighted

S.o.S and search-head pooling with job scheduler disabled

Splunk Employee
Splunk Employee

We just upgraded to 5.0.2 and I had a question about SOS.

We have 4 searchheads plus a jobs server. The jobs servers runs all scheduled searches. The 4 SH's have the scheduler pipeline disabled.

What will we be missing in SOS by doing this since no scheduled searches will be running on the SH's?

Highlighted

Re: S.o.S and search-head pooling with job scheduler disabled

Splunk Employee
Splunk Employee

The main issue is that S.o.S runs two lightweight scheduled searches to populate two lookups, one of which is heavily used in the app:

  • sos_splunk_instances_info -> splunk_instances_info.csv -> used to populate the "A glimpse of your Splunk instance" info panel in the Home view
  • sos_refresh_splunk_servers_cache -> splunk_servers_cache.csv -> used to populate the "Server to query" pulldown which exists in most views

At a minimum, you'll need to manually run those searches once to ensure that S.o.S discovers your topology. Given that you are in a search-head pool, you'll need to run this once per search-head in the pool.

Here's how you would run these searches manually:

  • Go to the flashtimeline within the context of the S.o.S app - http[s]:[splunkweb_hostname]:[splunkweb_port]/en-US/app/sos/flashtimeline
  • Run the following two searches:
    | savedsearch sos_refresh_splunk_servers_cache
    | savedsearch sos_splunk_instances_info | outputlookup splunk_instances_info

Note that you should also run these searches whenever a new indexer or pooled search-head is added, to help S.o.S discover it.

View solution in original post