Knowledge Management

Index Latency Summary

hartfoml
Motivator

I am calculating the index latency like this

index=firewall | eval diff = _indextime - _time

This is taking some time to run a 24 hour report as firewall logs are prolific.

Does anyone know if the latency for indexes is in a summary index anywhere?

Tags (2)
0 Karma
1 Solution

RicoSuave
Builder

Latencies are not stored anywhere. Your approach is correct, however you should be running this report in realtime using the all time (real time) dropdown option. This is to ensure that all incoming events are shown regardless of their extracted time stamp. Note that the latency measured here is affected by how old your events are when you expose them to Splunk. Alternatively, the SOS app includes a distributed indexing performance view that will show you realtime latency and is powered by this search

index=* OR index=_internal

| eval latency=round((_indextime - _time),2)
| eval seconds_elapsed=(time() - now())
| eval secs=if(seconds_elapsed<0,"1",seconds_elapsed)
| eval esize=((len(_raw)/1024))
| eventstats max(secs) AS seconds
| eventstats count AS ecount, sum(esize) AS sum_esize $type$
| stats last(ecount) AS "event count"
last(eval(ecount/seconds)) AS eps
last(eval(sum_esize/seconds)) AS KBps
min(latency) AS "minimum latency (seconds)"
avg(latency) AS avglat
max(latency) AS "maximum latency (seconds)"
min(_time) AS oldestTime
max(_time) AS newestTime $type$
| eval avglat=round(avglat,2)
| eval eps=round(eps,2)
| eval KBps=round(KBps,2)
| convert timeformat="%m/%d/%Y %H:%M:%S" ctime(newestTime)
| convert timeformat="%m/%d/%Y %H:%M:%S" ctime(oldestTime)
| rename newestTime AS "Time stamp of newest event"
oldestTime AS "Time stamp of oldest event"
avglat AS "average latency (seconds)"
eps AS "events per second"
KBps AS "indexing rate (KBps)"

Possible values for $type$: by index | by host | by source | by sourcetype | by splunk_server

You can modify this search to suit your needs.

View solution in original post

0 Karma

RicoSuave
Builder

Latencies are not stored anywhere. Your approach is correct, however you should be running this report in realtime using the all time (real time) dropdown option. This is to ensure that all incoming events are shown regardless of their extracted time stamp. Note that the latency measured here is affected by how old your events are when you expose them to Splunk. Alternatively, the SOS app includes a distributed indexing performance view that will show you realtime latency and is powered by this search

index=* OR index=_internal

| eval latency=round((_indextime - _time),2)
| eval seconds_elapsed=(time() - now())
| eval secs=if(seconds_elapsed<0,"1",seconds_elapsed)
| eval esize=((len(_raw)/1024))
| eventstats max(secs) AS seconds
| eventstats count AS ecount, sum(esize) AS sum_esize $type$
| stats last(ecount) AS "event count"
last(eval(ecount/seconds)) AS eps
last(eval(sum_esize/seconds)) AS KBps
min(latency) AS "minimum latency (seconds)"
avg(latency) AS avglat
max(latency) AS "maximum latency (seconds)"
min(_time) AS oldestTime
max(_time) AS newestTime $type$
| eval avglat=round(avglat,2)
| eval eps=round(eps,2)
| eval KBps=round(KBps,2)
| convert timeformat="%m/%d/%Y %H:%M:%S" ctime(newestTime)
| convert timeformat="%m/%d/%Y %H:%M:%S" ctime(oldestTime)
| rename newestTime AS "Time stamp of newest event"
oldestTime AS "Time stamp of oldest event"
avglat AS "average latency (seconds)"
eps AS "events per second"
KBps AS "indexing rate (KBps)"

Possible values for $type$: by index | by host | by source | by sourcetype | by splunk_server

You can modify this search to suit your needs.

0 Karma

glitchcowboy
Path Finder

By The Way, I left this one to run for quite a while and forgot about it. It chewed up 10G in /opt/splunk/var/run/splunk/dispatch and caused me a small headache. So don't leave it run too long:)

Interesting data though! thanks!

0 Karma

hartfoml
Motivator

Thanks Jeotron

I was looking for a summary to speed up the search. I need a report of the latency on one index for the last 24 hours so I am not interested in real time. I did see the search from SOS thanks

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...