Monitoring Splunk

Index age date gone high to rouge data

lukessi
Path Finder

I have two indexes that have done to really high day age, I got a feeling a few of the PC's have had issues and gone back to default date. 

I looked around and can't find a easy way to work out what events it was or the host, I have came close but want something to say here is the smoking gun. 

Cheers for any help as per normal.

Labels (2)
0 Karma

Richfez
SplunkTrust
SplunkTrust

Hi,

I think checking into the field `_indextime` and how it compares to `_time`.  One former is the time that the event was ingested, the other is the time Splunk found and "put into the event".

index=* 
| eval indextime = _indextime
| eval index_lag = abs(_indextime - _time)
| stats count as "Number of events", max(index_lag) as "Max", avg(index_lag) as "Avg", perc95(index_lag) as "95th Percentile", min(index_lag) as "Min" by host

 

Change the first line to match the index you are searching (wineventlog?  Windows?) and maybe add a sourcetype or whatever else. you need.

The second line - _indextime is hard to use "raw" because it's a special field.  So we just eval a new copy of it into a new field to use later.

The third line is the meat of calculating the difference between the two values.  We call it "index_lag"

The last line does your stats.   It gives you, by host, the number of events over whatever time period you have in the time picker, the largest index_lag for that host, the average index_lag, the 95th percentile of the lag for that host, and the minimum lag for it too.

That should

a) Give you a smoking gun

b) Also give you a lot more ammunition and knowledge about other things to search for in answers here if you need to do more.

Happy Splunking!

-Rich

0 Karma
Get Updates on the Splunk Community!

Developer Spotlight with Paul Stout

Welcome to our very first developer spotlight release series where we'll feature some awesome Splunk ...

State of Splunk Careers 2024: Maximizing Career Outcomes and the Continued Value of ...

For the past four years, Splunk has partnered with Enterprise Strategy Group to conduct a survey that gauges ...

Data-Driven Success: Splunk & Financial Services

Splunk streamlines the process of extracting insights from large volumes of data. In this fast-paced world, ...