topic Re: Monitor Splunk in Monitoring Splunk

Monitor Splunk

taka — Tue, 11 Jun 2024 06:30:13 GMT

I want to monitor Splunk Enterprise in a cluster environment. I monitor the Splunk infrastructure with Newrelic, and I also want to use the DMC health check item.

Where can I get the health check item other than by updating it? Also, please let me know if there are any other ways to monitor Splunk.

Re: Monitor Splunk

deepakc — Tue, 11 Jun 2024 06:41:30 GMT

You can start by having a look at the many API's endpoints available for the aspects of Splunk, and yes we mainly use the monitoring console for cluster -checks and health + others - which uses API's for checks

Have a look at the below API endpoints, as a starting point.

#Cluster Specific
https://docs.splunk.com/Documentation/Splunk/9.0.2/RESTREF/RESTcluster#cluster.2Fmanager.2Fhealth

#Other API endpoints

https://docs.splunk.com/Documentation/Splunk/9.0.2/RESTREF/RESTlist#MLOC

Re: Monitor Splunk

taka — Tue, 11 Jun 2024 08:23:08 GMT

Hello deepakc

Thank you for your immediate reply!
Do you have any prerequisites or concerns when implementing monitoring of that API endpoint?

Re: Monitor Splunk

deepakc — Tue, 11 Jun 2024 08:44:31 GMT

These would come to mind first - there's plenty more, you can explore the others and use them as you as you see fit.

1. Check the overall health

/services/cluster/manager/health

2. Check Cluster Status of the peers (Indexers

/services/cluster/manager/peers

3. Check the indexing status

/services/cluster/manager/indexes

4. Check the Replication and Search Factor status

/services/cluster/manager/status

You can also check the CM's resources (CPU/MEM etc)
5. Check Resource Utilisation on the CM

/services/server/status/resource-usage/hostwide

Re: Monitor Splunk

deepakc — Tue, 11 Jun 2024 10:01:40 GMT

I forgot to mention in terms of pre-reqs:

1. Newrelic should have some way of using API calls, you can use Splunk Tokens for API use and as a way of authentication - see below link for info

https://docs.splunk.com/Documentation/Splunk/9.2.1/Security/CreateAuthTokens