Index Latency Summary

[hartfoml]

I am calculating the index latency like this

index=firewall | eval diff = _indextime - _time

This is taking some time to run a 24 hour report as firewall logs are prolific.

Does anyone know if the latency for indexes is in a summary index anywhere?

[RicoSuave · Accepted solution]

Latencies are not stored anywhere. Your approach is correct, however you should be running this report in realtime using the all time (real time) dropdown option. This is to ensure that all incoming events are shown regardless of their extracted time stamp. Note that the latency measured here is affected by how old your events are when you expose them to Splunk. Alternatively, the SOS app includes a distributed indexing performance view that will show you realtime latency and is powered by this search

index=* OR index=_internal

| eval latency=round((_indextime - _time),2)
| eval seconds_elapsed=(time() - now())
| eval secs=if(seconds_elapsed<0,"1",seconds_elapsed)
| eval esize=((len(_raw)/1024))
| eventstats max(secs) AS seconds
| eventstats count AS ecount, sum(esize) AS sum_esize $type$
| stats last(ecount) AS "event count"
last(eval(ecount/seconds)) AS eps
last(eval(sum_esize/seconds)) AS KBps
min(latency) AS "minimum latency (seconds)"
avg(latency) AS avglat
max(latency) AS "maximum latency (seconds)"
min(_time) AS oldestTime
max(_time) AS newestTime $type$
| eval avglat=round(avglat,2)
| eval eps=round(eps,2)
| eval KBps=round(KBps,2)
| convert timeformat="%m/%d/%Y %H:%M:%S" ctime(newestTime)
| convert timeformat="%m/%d/%Y %H:%M:%S" ctime(oldestTime)
| rename newestTime AS "Time stamp of newest event"
oldestTime AS "Time stamp of oldest event"
avglat AS "average latency (seconds)"
eps AS "events per second"
KBps AS "indexing rate (KBps)"

Possible values for $type$: by index | by host | by source | by sourcetype | by splunk_server

You can modify this search to suit your needs.

[glitchcowboy]

By The Way, I left this one to run for quite a while and forgot about it. It chewed up 10G in /opt/splunk/var/run/splunk/dispatch and caused me a small headache. So don't leave it run too long:)

Interesting data though! thanks!

[hartfoml]

Thanks Jeotron

I was looking for a summary to speed up the search. I need a report of the latency on one index for the last 24 hours so I am not interested in real time. I did see the search from SOS thanks