Hi,
Our group needs to read data that is managed and stored in another Splunk in our company. The other splunk will have clusters, and a cluster-manager. It appears that there are two different ways to peer into indexers. The first is to add each indexer individually, entering a username/password for each connection. Not the optimal method, for obvious reasons. The second method appears to be to click on "Add a cluster to be searched", and then enter a "Master URI" and a "Secrety Key". I like the concept of this option, but I do have questions.
1) Does the query go through the cluster master, or just the authentication? Is it just one time, or for each query?
ANS: The search head queries cluster master to get generation ID
and list of search peers. Once it's get that, it'll directly contact search peers directly from there on. It gets the updated list frequently but not for every time a search is initiated.
2) What happens if the cluster master goes down?
ANS: The search head will continue to work with previous list of peers it got. However, if cluster master is down and some other failure or issues happens in the indexer cluster, SH may search on incomplete dataset. See this for more information on this: https://docs.splunk.com/Documentation/Splunk/7.2.5/Indexer/Whathappenswhenamasternodegoesdown
3) We need to setup filtering, so that only certain indexes are queried - is there any way to do this when connecting to a cluster? I'm assuming that it's using the admin account. Is there a way to have it connect using a different account?
ANS: You'd need an admin account/credentials to connect to indexer cluster. You should be able to restrict the indexes to be searched in your search head's authorize.conf for each role.