topic Re: Notification when indexes stop receiving data in Getting Data In

Notification when indexes stop receiving data

lavster — Wed, 19 Jul 2023 17:35:39 GMT

Hi, we’ve had a problem recently where data has stopped flowing to an index, and it’s a few days before we find out and then resolve. Does anyone know of a splunk 9.x feature or an add-on that you can use to monitor / alert when data stops for a set amount of time?

Re: Notification when indexes stop receiving data

meetmshah — Wed, 19 Jul 2023 17:56:30 GMT

TrackMe (https://splunkbase.splunk.com/app/4621) is the application that can help you.

Also, you can built custom saved searches which looks over the latest _time of the events and alert if the time difference between current time and latest(_time) is more than threashold. Threashold can be maintained in the lookup and called in the search. For example, create a lookup called acceptable_diff.csv with sample entries as below -

and run the below search -

| tstats latest(_time) as lastEventTime where index=* by index
| eval current_time=now()
| eval diff=current_time-lastEventTime
| lookup acceptable_diff.csv index as index OUTPUTNEW acceptable_diff as acceptable_diff
| eval diffWithAcceptableDiff=diff-acceptable_diff
| sort 0 - lastEventTime
| eval lastEventTime=strftime(lastEventTime,"%m/%d/%y %H:%M:%S")
| fields index lastEventTime diffWithAcceptableDiff acceptable_diff
| search diffWithAcceptableDiff>0

Feel free to accept the answer if it helps!

Re: Notification when indexes stop receiving data

sainag_splunk — Wed, 19 Jul 2023 19:03:20 GMT

Try something like this from the metrics log.
index=_internal source=*metrics.log* host=idx* series=yourindexname | stats latest(_time) as last_data_time by series | eval duration_seconds = now() - last_data_time | eval duration_human = strftime(duration_seconds, "%d days %H:%M:%S") | where duration_seconds > <your_duration_in_seconds>

Re: Notification when indexes stop receiving data

lavster — Thu, 20 Jul 2023 05:58:26 GMT

amazing thank you for this! i'll give this a go today