Monitoring Splunk

Index age date gone high to rouge data

lukessi
Path Finder

I have two indexes that have done to really high day age, I got a feeling a few of the PC's have had issues and gone back to default date. 

I looked around and can't find a easy way to work out what events it was or the host, I have came close but want something to say here is the smoking gun. 

Cheers for any help as per normal.

Labels (2)
0 Karma

Richfez
SplunkTrust
SplunkTrust

Hi,

I think checking into the field `_indextime` and how it compares to `_time`.  One former is the time that the event was ingested, the other is the time Splunk found and "put into the event".

index=* 
| eval indextime = _indextime
| eval index_lag = abs(_indextime - _time)
| stats count as "Number of events", max(index_lag) as "Max", avg(index_lag) as "Avg", perc95(index_lag) as "95th Percentile", min(index_lag) as "Min" by host

 

Change the first line to match the index you are searching (wineventlog?  Windows?) and maybe add a sourcetype or whatever else. you need.

The second line - _indextime is hard to use "raw" because it's a special field.  So we just eval a new copy of it into a new field to use later.

The third line is the meat of calculating the difference between the two values.  We call it "index_lag"

The last line does your stats.   It gives you, by host, the number of events over whatever time period you have in the time picker, the largest index_lag for that host, the average index_lag, the 95th percentile of the lag for that host, and the minimum lag for it too.

That should

a) Give you a smoking gun

b) Also give you a lot more ammunition and knowledge about other things to search for in answers here if you need to do more.

Happy Splunking!

-Rich

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Community Content Calendar, September edition

Welcome to another insightful post from our Community Content Calendar! We're thrilled to continue bringing ...

Splunkbase Unveils New App Listing Management Public Preview

Splunkbase Unveils New App Listing Management Public PreviewWe're thrilled to announce the public preview of ...

Leveraging Automated Threat Analysis Across the Splunk Ecosystem

Are you leveraging automation to its fullest potential in your threat detection strategy?Our upcoming Security ...