Splunk Search

timechart Index Time vs Raw Log's Time

morethanyell
Contributor

Hi,

This search string have helped us a lot during investigation. It paints a timechart / graphicall representation of when there was a dip in ingestion together with the log's raw time. Since it's using tsats, it's also fast.

| tstats min(_indextime) as it count where index=my_index by _time span=1ms 
| eval _time = it 
| fields - it 
| timechart span=5m sum(count) as it 
| join _time 
    [| tstats count as raw where index=my_index by _time span=5m ] 
| fillnull value=0

Here's how the graph looks like (set linechart Y to Log)
alt text

As you can see, the blue line dipped (index-time) and yellow line (log's raw time) shows that logs that did not ingest during the time the index-time dipped have already caught up.

Since this search is using min(_indextime) and is joining it in _time, there's no guarantee that it's 100% accurate. However, I can confidently say that it's almost always reliable.

If the two lines are overlapping with oneother (appearing almost as though there's only one line), that means that the logs are ingested almost instantaneously.

0 Karma

woodcock
Esteemed Legend

You should check out Meta Woot!:
https://splunkbase.splunk.com/app/2949/

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!