Splunk Search

timechart Index Time vs Raw Log's Time



This search string have helped us a lot during investigation. It paints a timechart / graphicall representation of when there was a dip in ingestion together with the log's raw time. Since it's using tsats, it's also fast.

| tstats min(_indextime) as it count where index=my_index by _time span=1ms 
| eval _time = it 
| fields - it 
| timechart span=5m sum(count) as it 
| join _time 
    [| tstats count as raw where index=my_index by _time span=5m ] 
| fillnull value=0

Here's how the graph looks like (set linechart Y to Log)
alt text

As you can see, the blue line dipped (index-time) and yellow line (log's raw time) shows that logs that did not ingest during the time the index-time dipped have already caught up.

Since this search is using min(_indextime) and is joining it in _time, there's no guarantee that it's 100% accurate. However, I can confidently say that it's almost always reliable.

If the two lines are overlapping with oneother (appearing almost as though there's only one line), that means that the logs are ingested almost instantaneously.

0 Karma

Esteemed Legend

You should check out Meta Woot!:

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!