All Apps and Add-ons

Splunk Health Check Overview: Can we add search heads as search peers to monitor them?

Path Finder

We have a clustered environment running Splunk 6.1.7 with 3 Search Heads, 3 Search Peers, and 2 Heavy Forwarders with the Health Check Overview app installed in the master node. If we want to gather information from all hosts, should we add all the instances as search peers, or is it not recommended to do this? What would be the consequences, sending bundles to all hosts?

0 Karma

Splunk Employee
Splunk Employee

You should not need to add them as search peers if you have the app installed on the master node. Splunk 6.2 offers several rest endpoints that the app takes advantage of to assign server roles but for 6.1.x you will need to enable the server_lookup_v6_1 saved search to populate the host information. A new version of the application is currently in the works and will be released shortly.

0 Karma

Splunk Employee
Splunk Employee

You will want to setup the search heads to forward the logs to the indexers via outputs.conf and restart. Once you add that you will need to re-run the server_lookup_v6_1. Alternatively, you can set them up as search peers but it is best practice to have all components log their internal logs to the indexing tier.

0 Karma

Path Finder

I've revised the forwarding of the internal logs and is configured indeed...

the outputs.conf in our searches is nailed with the outputs covered in http://docs.splunk.com/Documentation/Splunk/6.3.1/DistSearch/Forwardsearchheaddata

In fact, a search query on _internal shows all hosts in our cluster environment. Is crucial for us to see the the activity on any search heads. Editing the server query I see that

|inputlookup all_servers.csv

only returns our indexers

To see what is happening I have playing with the 6.1 lookup and I see that my searchers are being discarded because the join with the rest call is not returning values for the search heads.

This query

index=_internal sourcetype=splunkd component=ServerRoles role!=license_slave* role!=search_peer* |rex field=role "(?<role>\S+)\." |dedup host role | join host [|rest splunk_server=* /services/server/info | rename serverName AS host | fields host guid version] | rename server_role AS role | table host role guid version

returns only the indexers

this one

index=_internal sourcetype=splunkd component=ServerRoles role!=license_slave* role!=search_peer* | table host role guid version

return them all

0 Karma

Path Finder

When I execute the 6.1 lookup I can see the master node and the search peers. Since the lookup table is used in many dashboards in dropdown menus to select the servers I cannot see searches or users in other search heads. For example in Searches - Search activity I can only see the activity on the master node.

0 Karma

Splunk Employee
Splunk Employee

Are your search heads forwarding their logs to the indexing tier?

0 Karma

Path Finder

nope, We are not forwarding any internal logs

0 Karma