Alerting

Logging frequency of my index,sourcetype and host

thippeshaj
Explorer

Hi Splunkers,

How do I calculate the logging frequency of my index=xxx sourcetype=yyy host=zzz?

Explanation: I have a different set of logs which sends logs with different frequency, some of them send every minute/hour and some of them send the logs once a day. so basically logging frequency is not fixed, it's dynamic.

I'm trying to find out a way to alert if a particular index=xxx sourcetype=yyy host=zzz stops sending logs, I want a dynamic way of calculating the frequency threshold wherein I can say (now()-last_event_time) > threshold, I don't want to use something like which is basically find the difference between the last event time VS the current and some random threshold.

I want Splunk to tell the ideal threshold for my index,sourcetype and host combination.
For example; A particular logs from index=a host=b sourceype=c logs once in a day, so here I want Splunk to tell an ideal threshold I can use ( maybe ~ 1day or ~ 24 hours) as the threshold to set an alert.
another example, a particular logs from index=g sourcetype=h host=i logs every 4.5/5 hours, so here I want Splunk to tell an ideal threshold I can use ( maybe ~ 4H or ~ 4.5h or 5h or xh) as the threshold to set an alert.

so using this I can set an alert like (now()-last_event_time) > threshold

Thanks in advance.

Happy Splunking.

Labels (1)
0 Karma

amehlmann
Engager

Hi,

 

you can calculate the average timespan between events using

| tstats count as event_count, latest(_time) as latest_time, earliest(_time) as earliest_time by host,index
| eval total_time_spans = latest_time - earliest_time
| eval average_time_span = total_time_spans / (event_count - 1)
| stats avg(average_time_span) as avg_time_span by host,index

 

But beware since this makes only sense if you have regular reporting hosts/indexes. this will not work if one host e.g. sends 1k events once a day.

0 Karma

MKozanic
Path Finder

Hi @thippeshaj ,

Just wondering if you worked out a solution for this?

I'm currently looking for something similar to be able to configure alerts based on previous indexing frequency

0 Karma

dmuralidaran_sp
Splunk Employee
Splunk Employee

How about configuring an alert to see if you get new data every interval according to the configuration for each host machine?

0 Karma
Get Updates on the Splunk Community!

New Case Study Shows the Value of Partnering with Splunk Academic Alliance

The University of Nevada, Las Vegas (UNLV) is another premier research institution helping to shape the next ...

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...