Hi All ,
I have developed a mini POC to look out for Splunk End to End Monitoring . The POC will be triggered if there is a missing log source being reported in the splunk alert. Below are my checks and i would like to know that whether i have missed any checks ?
Main Query : Splunk Query for missing log sources. This will trigger the below steps:
1. Splunk Connection to Search Head
1.a If splunk connection fails then check for network connection to Search head instance by a 'ping', followed by a health check on ports and services.
2. If connection is successfull, Splunk Query to check whether all indexers are reporting for last say 60 mins.
2.a if some of indexers are not reporting then, check for network connection to indexers with a ping followed by a health check on ports and services.
3. If connection is successfull , then Splunk query to check for Blocked Queues at Indexer level
4. Splunk Query to check for Missing forwarder.
5. If missing forwarder results, then check for forwarder availability with a ping, followed by a check on splunk socket connection and health check on ports and services.
6. Splunk Query to check for data throttling at forwarder level.
These are the checks that i have implemented which might cause a missing log source. Checks are only within Splunk Infra.
Please let me know if i have missed any checks
