Deployment Architecture

Watching the watchmen: how to monitor splunkd from the command line

robgarner
Path Finder

Hi -

I would like to monitor the status of a linux-based splunkd configured as a heavy forwarder from an external system (nagios, custom scripts, etc). The only visibility I'm aware of regarding the state of the daemon from the linux cli is 'splunk status splunkd' which just tells me if the daemon is running or not. Is there any way for splunkd to report, fer instance, how long its been running and the number of connections accepted in the last minutes ?

Thanks,
-Rob

0 Karma
1 Solution

koshyk
Super Champion

There are multiple ways you can do
1. Looking into the logs. We make use of "metrics.log" and use "tcpin_queue" to measure the data transmission. There are multiple components in this log which you can make use of. This would update frequently and based on your logic ensure nagios monitor it every 10-15mins and if there is no update then there is a problem.
2. REST api calls - you need to have a API enabled user within Splunk for this. Based on your endpoint you fire your REST call from your monitoring system especially the introspection endpoint. This contains all information for any monitoring. Just go into the level of detail you require. (eg: curl -k -u admin:changeme https://localhost:8089/services/server/status/resource-usage/splunk-processes)

View solution in original post

0 Karma

koshyk
Super Champion

There are multiple ways you can do
1. Looking into the logs. We make use of "metrics.log" and use "tcpin_queue" to measure the data transmission. There are multiple components in this log which you can make use of. This would update frequently and based on your logic ensure nagios monitor it every 10-15mins and if there is no update then there is a problem.
2. REST api calls - you need to have a API enabled user within Splunk for this. Based on your endpoint you fire your REST call from your monitoring system especially the introspection endpoint. This contains all information for any monitoring. Just go into the level of detail you require. (eg: curl -k -u admin:changeme https://localhost:8089/services/server/status/resource-usage/splunk-processes)

0 Karma

robgarner
Path Finder

Very helpful, many thanks !

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...