Alerting

How to create an alert based on these data assurance scenarios?

indianhans
Engager

Hi All,

I am seeking some thoughts to implement data assurance. I wish to build an alerting mechanism for following circumstances :

1) If a source stops sending the logs.
2) If there is any delay in Log indexing.
** 3) If the log format at source is changed. / Log pattern is changed.

Is there any easy solution to alert on above scenarios, especially Case 3 (Log pattern change).

Regards
Rishi

0 Karma
1 Solution

woodcock
Esteemed Legend

For the first 2, you should be able to find existing searches in your MC ( Settings -> Montioring Console ). Also for the 2nd one, you can track avg(_indextime - _time) over time by index, host, and sourcetype. For the last one, you need to track values(punct) or dc(punct) or | cluster | stats count over time by index, host, and sourcetype.

The cluster command is what drives the patterns tab:
http://docs.splunk.com/Documentation/Splunk/6.5.1/SearchReference/Cluster

View solution in original post

woodcock
Esteemed Legend

For the first 2, you should be able to find existing searches in your MC ( Settings -> Montioring Console ). Also for the 2nd one, you can track avg(_indextime - _time) over time by index, host, and sourcetype. For the last one, you need to track values(punct) or dc(punct) or | cluster | stats count over time by index, host, and sourcetype.

The cluster command is what drives the patterns tab:
http://docs.splunk.com/Documentation/Splunk/6.5.1/SearchReference/Cluster

indianhans
Engager

Thanks for your valuable inputs. I am already trying my hands on "Punct / Cluster". Can you please suggest some other ways, so that I can compare the results and accuracy ?

Thanks again. 🙂

0 Karma

woodcock
Esteemed Legend

Using punct is quick and dirty, using cluster is highly configurable and nuanced. Those are your options.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...