Alerting

Longest period of time without any events in an index

chalak
Path Finder

Ultimate goal is to find out what is the longest period of time without any event in an index within last month ( and to this across all the indexes automatically, without necessity to specify the indexes within any query)

Currently I have two suggestions:

  1. The first one works across all indexes:

    | metasearch index=* | transaction maxevents=2 index host | stats max(duration) AS current_max_delay by index, host

  2. The second one works only for one
    index (not helpful, placing here just for possible inspiration):

    index=INDEX_A | delta time AS AtimeDelta | eval Apossitive_delta = (abs(AtimeDelta)/3600) | stats max(Apossitive_delta) as A_MaxDelta
    FIRST question: How to solve a situation when there might be just one event in last month, where I would like to identify not the delta between events ( as there are no other), but between the event and the current time?

SECOND question: There might be many indexes and large quantity of data. Is there any way how to make this more efficient?

Many thanks for any suggestions.

0 Karma

niketn
Legend

@adonio, I would try the following where rest API gives the list or all non internal indexes and tstats finds the duration since last indexes event per index.

| tstats latest(_time) as LatestEventTime where 
    [| rest /services/data/indexes count=0 
    | search title!="_*"
    | rename title as index 
    | table index] by index
| eval duration=now()-LatestEventTime 
| append 
    [| rest /services/data/indexes count=0 
    | search title!="_*" 
    | rename title as index 
    | fields index 
    | addinfo 
    | rename info_min_time as LatestEventTime 
    | fields index LatestEventTime 
    | eval duration= now()-LatestEventTime] 
| dedup index 
| eval duration=tostring(duration,"duration") 
| eval duration=if(match(duration,"\+"),replace(duration,"(\d+)\+(\d+)\:(\d+)\:(\d+)","\1 Day \2 Hr \3 Min \4 Secs"),replace(duration,"(\d+)\:(\d+)\:(\d+)","\1 Hr \2 Min \3 Secs"))

PS: | search title!="_*" can be removed for querying all indexes including Splunk's internal indexes.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

chalak
Path Finder

Hi niketnilay, thank you for the suggestion. Similarly with the previous comment from andonio, the challenge with this is that it will take the last event in the index, which could have happened a second ago. But before that there could have been two weeks period without an event. Would you have a clue how to effectively find out the longest period of time between any events within the given time-frame (a month) with one additional corner case (when there would be completely empty index for a month and just 10 events at the beginning of the month with the same timestamp? In that case the time delta would be 0 between the events, but effectively it would be 29days (which would be correctly pick-up by your search query)

0 Karma

adonio
Ultra Champion

try this in the last 30 days

| tstats max(_time) as last_event where index=* by index
| eval now_time = now()
| eval period_mot_indexed = now_time - last_event

chalak
Path Finder

Hi Adonio, thank you for the suggestion. I have tried this before, the challenge with this is that it will take the last event in the index, which could have happened a second ago. But before that there could have been two weeks period without an event. Would you have a clue how to effectively find out the longest period of time between any events within the given time-frame (a month) with one additional corner case (when there would be completely empty index for a month and just 10 events at the beginning of the month with the same timestamp?

0 Karma
Get Updates on the Splunk Community!

Splunk Classroom Chronicles: Training Tales and Testimonials

Welcome to the "Splunk Classroom Chronicles" series, created to help curious, career-minded learners get ...

Access Tokens Page - New & Improved

Splunk Observability Cloud recently launched an improved design for the access tokens page for better ...

Stay Connected: Your Guide to November Tech Talks, Office Hours, and Webinars!

🍂 Fall into November with a fresh lineup of Community Office Hours, Tech Talks, and Webinars we’ve ...