How to create an alert based on these data assurance scenarios?

[indianhans]

Hi All,

I am seeking some thoughts to implement data assurance. I wish to build an alerting mechanism for following circumstances :

1) If a source stops sending the logs.
2) If there is any delay in Log indexing.
** 3) If the log format at source is changed. / Log pattern is changed.

Is there any easy solution to alert on above scenarios, especially Case 3 (Log pattern change).

Regards
Rishi

[woodcock · Accepted solution]

For the first 2, you should be able to find existing searches in your MC ( Settings -> Montioring Console ). Also for the 2nd one, you can track avg(_indextime - _time) over time by index, host, and sourcetype. For the last one, you need to track values(punct) or dc(punct) or | cluster | stats count over time by index, host, and sourcetype.

The cluster command is what drives the patterns tab:
http://docs.splunk.com/Documentation/Splunk/6.5.1/SearchReference/Cluster

[indianhans]

Thanks for your valuable inputs. I am already trying my hands on "Punct / Cluster". Can you please suggest some other ways, so that I can compare the results and accuracy ?

Thanks again. 🙂

[woodcock]

Using punct is quick and dirty, using cluster is highly configurable and nuanced. Those are your options.