We have a clustered environment running Splunk 6.1.7 with 3 Search Heads, 3 Search Peers, and 2 Heavy Forwarders with the Health Check Overview app installed in the master node. If we want to gather information from all hosts, should we add all the instances as search peers, or is it not recommended to do this? What would be the consequences, sending bundles to all hosts?
You should not need to add them as search peers if you have the app installed on the master node. Splunk 6.2 offers several rest endpoints that the app takes advantage of to assign server roles but for 6.1.x you will need to enable the server_lookup_v6_1 saved search to populate the host information. A new version of the application is currently in the works and will be released shortly.
You will want to setup the search heads to forward the logs to the indexers via outputs.conf and restart. Once you add that you will need to re-run the server_lookup_v6_1. Alternatively, you can set them up as search peers but it is best practice to have all components log their internal logs to the indexing tier.
I've revised the forwarding of the internal logs and is configured indeed...
the outputs.conf in our searches is nailed with the outputs covered in http://docs.splunk.com/Documentation/Splunk/6.3.1/DistSearch/Forwardsearchheaddata
In fact, a search query on _internal shows all hosts in our cluster environment. Is crucial for us to see the the activity on any search heads. Editing the server query I see that
only returns our indexers
To see what is happening I have playing with the 6.1 lookup and I see that my searchers are being discarded because the join with the rest call is not returning values for the search heads.
index=_internal sourcetype=splunkd component=ServerRoles role!=license_slave* role!=search_peer* |rex field=role "(?<role>\S+)\." |dedup host role | join host [|rest splunk_server=* /services/server/info | rename serverName AS host | fields host guid version] | rename server_role AS role | table host role guid version
returns only the indexers
index=_internal sourcetype=splunkd component=ServerRoles role!=license_slave* role!=search_peer* | table host role guid version
return them all
When I execute the 6.1 lookup I can see the master node and the search peers. Since the lookup table is used in many dashboards in dropdown menus to select the servers I cannot see searches or users in other search heads. For example in Searches - Search activity I can only see the activity on the master node.