topic Re: How to monitor search head cluster from a monitoring console? in Deployment Architecture

How to monitor search head cluster from a monitoring console?

vinaypradhan — Tue, 29 Oct 2019 20:11:53 GMT

Hi,
I have a clustered environment where I have 1 indexer master/license master, 1 search head deployer, 3 search heads in search head cluster and 2 indexers in an indexer cluster.
I have set up a monitoring console on the license master and changed it to distributed mode.
I can see my indexers there, but I don't see my search heads.

I followed the documentation and went to Settings -> Distributed Search -> Search Peers and tried doing add new search peer and provided my search head URL
https://xxxxx1:8089

I added all my search heads, but when I add it, I see that the cluster label shows as indexercluster1 - that's my indexer cluster label.

Why is my search head showing as an indexer cluster member when I add it here?
Also, the replication status gets set to Initial when I add it and then changes to Successful.
What is it replicating?

And even after this, when I check the topology under search heads it's not showing any of the above machines that I added.
I presume it's considering them as indexers as its showing the indexer cluster label.

How do I fix this so that I can monitor my search heads and the search head deployer too through this monitoring console?
Any help is greatly appreciated.

Re: How to monitor search head cluster from a monitoring console?

ivanreis — Tue, 29 Oct 2019 21:55:44 GMT

you have to edit the server roles at MC and setup the search head hosts to search head.
Go to Management Console/Settings/Forwarders/General Setup/Actions/Edit and change to "search head" role

As you are working on an indexer cluster environment, the data is being replicated to the entire indexer cluster, so if one of indexers cluster peers went down, you are still able to search the data.
I recommend for you to read this document to understand how indexer replication works
https://docs.splunk.com/Documentation/Splunk/7.3.2/Indexer/Aboutclusters

Re: How to monitor search head cluster from a monitoring console?

vinaypradhan — Wed, 30 Oct 2019 14:52:43 GMT

hi IvanReis, thanks for your reply. I do understand indexer cluster replication. but when i go to Settings -> Distributed Search -> Search Peers and add my search head there, it shows my indexer cluster label against it and shows replication is Initial and after some time successful - i dont want any of my indexed data to be replicated on my search heads. the sole reason i am adding my search head here, is to be able to get my search head in the monitoring console to show up as a search head, but even after adding it here, it doesnt show up in my monitoring console as a search head

Re: How to monitor search head cluster from a monitoring console?

ivanreis — Wed, 30 Oct 2019 23:34:54 GMT

Per my understanding, it seems your search head is setup as indexer at management console. You have to change the search head server role to only search head. If the server role is not properly defined at MC, the server will be added to other role, in some cases you have to manually changed it.

Management Console/Settings/Forwarders/General Setup/Actions/Edit and change to "search head" role

After run this configuration, save it and restart the MC and check if the server is setup to "search head" role.

It is considered a best practice to forward all search head internal data to the search peer (indexer) layer. Check the document below
https://docs.splunk.com/Documentation/Splunk/7.3.2/DistSearch/Forwardsearchheaddata
Make sure your search head clusters is f

Re: How to monitor search head cluster from a monitoring console?

woodcock — Wed, 30 Oct 2019 23:43:39 GMT

Do/check these on MC:
Go to Search peers and ensure that ALL Splunk infrastructure nodes are peers. When you peer the CM, the Indexers should peer in, but if not, add those, too.
Go to Monitoring Console -> Setup -> General Setup and select Distributed Mode then edit each peer to manually assign the correct roles. Click Apply and then PROFIT!!!

Re: How to monitor search head cluster from a monitoring console?

vinaypradhan — Tue, 05 Nov 2019 14:15:35 GMT

thank you all, i was able to follow woodcock's suggestion and get my search heads in the monitoring console.

Re: How to monitor search head cluster from a monitoring console?

SamHTexas — Thu, 01 Apr 2021 18:11:40 GMT

Mr. Woodcock, I have 2 MC in distributed mode - only 1 showing peers. 2 MC in standalone mode no peers defined. It is something I inherited. Does this look like over doing it? Too much resources used for same reasons? Please advise.