Deployment Architecture

Watching the watchmen: how to monitor splunkd from the command line

robgarner
Path Finder

Hi -

I would like to monitor the status of a linux-based splunkd configured as a heavy forwarder from an external system (nagios, custom scripts, etc). The only visibility I'm aware of regarding the state of the daemon from the linux cli is 'splunk status splunkd' which just tells me if the daemon is running or not. Is there any way for splunkd to report, fer instance, how long its been running and the number of connections accepted in the last minutes ?

Thanks,
-Rob

0 Karma
1 Solution

koshyk
Super Champion

There are multiple ways you can do
1. Looking into the logs. We make use of "metrics.log" and use "tcpin_queue" to measure the data transmission. There are multiple components in this log which you can make use of. This would update frequently and based on your logic ensure nagios monitor it every 10-15mins and if there is no update then there is a problem.
2. REST api calls - you need to have a API enabled user within Splunk for this. Based on your endpoint you fire your REST call from your monitoring system especially the introspection endpoint. This contains all information for any monitoring. Just go into the level of detail you require. (eg: curl -k -u admin:changeme https://localhost:8089/services/server/status/resource-usage/splunk-processes)

View solution in original post

0 Karma

koshyk
Super Champion

There are multiple ways you can do
1. Looking into the logs. We make use of "metrics.log" and use "tcpin_queue" to measure the data transmission. There are multiple components in this log which you can make use of. This would update frequently and based on your logic ensure nagios monitor it every 10-15mins and if there is no update then there is a problem.
2. REST api calls - you need to have a API enabled user within Splunk for this. Based on your endpoint you fire your REST call from your monitoring system especially the introspection endpoint. This contains all information for any monitoring. Just go into the level of detail you require. (eg: curl -k -u admin:changeme https://localhost:8089/services/server/status/resource-usage/splunk-processes)

0 Karma

robgarner
Path Finder

Very helpful, many thanks !

0 Karma
Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...