Monitoring Splunk

Splunk End TO End Monitoring ?

lohit
Path Finder

Hi All ,

I have developed a mini POC to look out for Splunk End to End Monitoring . The POC will be triggered if there is a missing log source being reported in the splunk alert. Below are my checks and i would like to know that whether i have missed any checks ?

Main Query : Splunk Query for missing log sources. This will trigger the below steps:
1. Splunk Connection to Search Head
1.a If splunk connection fails then check for network connection to Search head instance by a 'ping', followed by a health check on ports and services.
2. If connection is successfull, Splunk Query to check whether all indexers are reporting for last say 60 mins.
2.a if some of indexers are not reporting then, check for network connection to indexers with a ping followed by a health check on ports and services.
3. If connection is successfull , then Splunk query to check for Blocked Queues at Indexer level
4. Splunk Query to check for Missing forwarder.
5. If missing forwarder results, then check for forwarder availability with a ping, followed by a check on splunk socket connection and health check on ports and services.
6. Splunk Query to check for data throttling at forwarder level.

These are the checks that i have implemented which might cause a missing log source. Checks are only within Splunk Infra.

Please let me know if i have missed any checks

Tags (1)
0 Karma

lohit
Path Finder

I have this done and deployed 🙂

0 Karma

dmerritt77
New Member

I'm trying to develop something similar, would love to see what you have so far if possible?

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!