We are running S.o.S - Splunk on Splunk in a search head clustering environment. Is it possible to get the resource utilization information from all search heads to be searchable from a single search head using the S.o.S add-on for Linux? Right now, I have to log in to Splunk Web for each search head to get the data for that search head, (i.e. I have to log in to searchhead1:8000 to view info for searchhead1).
Where does your search-head maintain its data , and is that location searchable from the other search heads ? Do you have an outputs.conf on your search-head that redirects the outputs to an indexer ?
The default behavior of Splunk is to maintain (index) data locally. So your search head in the cluster, is also an indexer for local data only. The issue you see is due to the fact that the search head you are logging into, does not have access to the data indexed on other search head nodes in the cluster.
In our case, we forward all data from a search head off to an indexer, where it is indexed, and maintained. These indexers are searchable from all the search head nodes in the cluster, and the above problem is avoided.
Our search heads write data locally right now, because we don't know how to configure it any other way. Ideally, we would write send this data to the index cluster to be indexed with the rest of our data. It sounds like we need to utilize an outputs.conf file to forward the locally indexed data to the index cluster.
Which outputs.conf file would we use, and would this have any other potential impacts to the system?