In a Splunk dashboard, how do you track data ingestion within a certain time?

[maryamchar]

Hello,

On a Splunk dashboard, Is there a way to show when data was ingested, stored, and analyzed? I'm trying to build a dashboard to show the following:

1. Source time
2. Arrival time for data
3. Fetch time when it was available
4. ingested time

I want to prove that it all happened within 10 minutes, and then show that proof in a dashboard. I'm using Splunk Enterprise Search and reporting.

Could you please help me write a query for that or if there are any examples that would be great. Thank you for help in advance!

[Vijeta · Accepted solution]

Source time is your _time field in the index
Ingested time/arrival time/fetched time all should be same that is when the data got ingested it is available for search- the field is _indextime

[maryamchar]

This is what i have so far, however, the time is in seconds now, and i don't understand what that time means. Is there a way to change the seconds to (H:M:S). Again thank you for help!

Source = " " index= " " host = " "
| eval delay_sec=_indextime-_time
| timechart min(delay_sec) avg(delay_sec) max(delay_sec) by source

[Vijeta]

The value delay_sec should be in seconds , you don't need to convert it in H:M:S format. If you want to display in minutes, you can divide it by 60

[maryamchar]

I tried dividing by 60 it didn't show any results.

[Vijeta]

what is the value it shows you before dividing by 60?

[Vijeta]

Ideally you should be getting the delay in seconds and you do not need any conversion if the delay is usually a few seconds.

[maryamchar]

Here are all the values i'm getting after running this query:
Source = " " index= " " host = " "
| eval delay_sec=_indextime-_time
| timechart min(delay_sec) avg(delay_sec) max(delay_sec) by source

avg(delay_sec): 19720226
max(delay_sec): 19936226
min(delay_sec): 19504226

I'm trying to check how long did this process takes after ingested data, and when it shows on my search by having a dashboard. But the values above is not really telling me much. Is there a better way ? Sorry for asking a lot of questions

[Vijeta]

Use the query like this-

Source = " " index= " " host = " " 
| eval delay_sec=_indextime-_time 
| timechart min(delay_sec)  as min_delay avg(delay_sec) as avg_delay max(delay_sec) as max_delay by source
| eval avg_delay=strftime(avg_delay, "%Y-%m-%d %H:%M:%S")
| eval min_delay=strftime(avg_delay, "%Y-%m-%d %H:%M:%S")
| eval max_delay=strftime(avg_delay, "%Y-%m-%d %H:%M:%S")

[maryamchar]

I got the same exact values in seconds still

[Vijeta]

You actually dont need to use strftime. The delay is the timedifference in seconds between the index time and the sourcetime.
By looking at the values of result it looks like there is a huge delay as 19720226 seconds which is around 350 days. Can you share the value of _time from your logs

[maryamchar]

_time: 2018-01 I'm testing old data but this is just for testing purpose and learning. I'm just trying to see if it's possible to do it the way i have it already.
Also, min, avg, max what does those exactly tell me in this case?

[Vijeta]

Well min here tells you minimum difference between indextime and _time for a particular source, avg will give you an average delay, and max will give you maximum delay between index time and _time field for a given source.
Also the value of _time depends on the datetimestamp configured for your logs. Ideally it should be date timestamp inside your logs but sometimes it is set as Current date and time in which case index time and _time would be same.
The best way to check the different between a set of ecents would be just to display logs with _time and _indextime value

index = <yourindex> sourcetype=<yoursourcetype>| eval time=strftime(_time,"%Y-%m-%d %H:%M:%S")| eval  indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S")| table _raw time indextime

[maryamchar]

Thank you!