Splunk End TO End Monitoring ?

lohit — Thu, 18 Jun 2015 10:51:00 GMT

Hi All ,

I have developed a mini POC to look out for Splunk End to End Monitoring . The POC will be triggered if there is a missing log source being reported in the splunk alert. Below are my checks and i would like to know that whether i have missed any checks ?

Main Query : Splunk Query for missing log sources. This will trigger the below steps:
1. Splunk Connection to Search Head
1.a If splunk connection fails then check for network connection to Search head instance by a 'ping', followed by a health check on ports and services.
2. If connection is successfull, Splunk Query to check whether all indexers are reporting for last say 60 mins.
2.a if some of indexers are not reporting then, check for network connection to indexers with a ping followed by a health check on ports and services.
3. If connection is successfull , then Splunk query to check for Blocked Queues at Indexer level
4. Splunk Query to check for Missing forwarder.
5. If missing forwarder results, then check for forwarder availability with a ping, followed by a check on splunk socket connection and health check on ports and services.
6. Splunk Query to check for data throttling at forwarder level.

These are the checks that i have implemented which might cause a missing log source. Checks are only within Splunk Infra.

Please let me know if i have missed any checks

Re: Splunk End TO End Monitoring ?

dmerritt77 — Wed, 26 Aug 2015 18:29:43 GMT

I'm trying to develop something similar, would love to see what you have so far if possible?

Re: Splunk End TO End Monitoring ?

lohit — Thu, 27 Aug 2015 07:35:53 GMT

I have this done and deployed 🙂

topic Re: Splunk End TO End Monitoring ? in Monitoring Splunk

Splunk End TO End Monitoring ?

Re: Splunk End TO End Monitoring ?

Re: Splunk End TO End Monitoring ?