Monitoring Splunk

Index age date gone high to rouge data

lukessi
Path Finder

I have two indexes that have done to really high day age, I got a feeling a few of the PC's have had issues and gone back to default date. 

I looked around and can't find a easy way to work out what events it was or the host, I have came close but want something to say here is the smoking gun. 

Cheers for any help as per normal.

Labels (2)
0 Karma

Richfez
SplunkTrust
SplunkTrust

Hi,

I think checking into the field `_indextime` and how it compares to `_time`.  One former is the time that the event was ingested, the other is the time Splunk found and "put into the event".

index=* 
| eval indextime = _indextime
| eval index_lag = abs(_indextime - _time)
| stats count as "Number of events", max(index_lag) as "Max", avg(index_lag) as "Avg", perc95(index_lag) as "95th Percentile", min(index_lag) as "Min" by host

 

Change the first line to match the index you are searching (wineventlog?  Windows?) and maybe add a sourcetype or whatever else. you need.

The second line - _indextime is hard to use "raw" because it's a special field.  So we just eval a new copy of it into a new field to use later.

The third line is the meat of calculating the difference between the two values.  We call it "index_lag"

The last line does your stats.   It gives you, by host, the number of events over whatever time period you have in the time picker, the largest index_lag for that host, the average index_lag, the 95th percentile of the lag for that host, and the minimum lag for it too.

That should

a) Give you a smoking gun

b) Also give you a lot more ammunition and knowledge about other things to search for in answers here if you need to do more.

Happy Splunking!

-Rich

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...