Monitoring Splunk

Index age date gone high to rouge data

lukessi
Path Finder

I have two indexes that have done to really high day age, I got a feeling a few of the PC's have had issues and gone back to default date. 

I looked around and can't find a easy way to work out what events it was or the host, I have came close but want something to say here is the smoking gun. 

Cheers for any help as per normal.

Labels (2)
0 Karma

Richfez
SplunkTrust
SplunkTrust

Hi,

I think checking into the field `_indextime` and how it compares to `_time`.  One former is the time that the event was ingested, the other is the time Splunk found and "put into the event".

index=* 
| eval indextime = _indextime
| eval index_lag = abs(_indextime - _time)
| stats count as "Number of events", max(index_lag) as "Max", avg(index_lag) as "Avg", perc95(index_lag) as "95th Percentile", min(index_lag) as "Min" by host

 

Change the first line to match the index you are searching (wineventlog?  Windows?) and maybe add a sourcetype or whatever else. you need.

The second line - _indextime is hard to use "raw" because it's a special field.  So we just eval a new copy of it into a new field to use later.

The third line is the meat of calculating the difference between the two values.  We call it "index_lag"

The last line does your stats.   It gives you, by host, the number of events over whatever time period you have in the time picker, the largest index_lag for that host, the average index_lag, the 95th percentile of the lag for that host, and the minimum lag for it too.

That should

a) Give you a smoking gun

b) Also give you a lot more ammunition and knowledge about other things to search for in answers here if you need to do more.

Happy Splunking!

-Rich

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...