Monitoring Splunk

Index age date gone high to rouge data

lukessi
Path Finder

I have two indexes that have done to really high day age, I got a feeling a few of the PC's have had issues and gone back to default date. 

I looked around and can't find a easy way to work out what events it was or the host, I have came close but want something to say here is the smoking gun. 

Cheers for any help as per normal.

Labels (2)
0 Karma

Richfez
SplunkTrust
SplunkTrust

Hi,

I think checking into the field `_indextime` and how it compares to `_time`.  One former is the time that the event was ingested, the other is the time Splunk found and "put into the event".

index=* 
| eval indextime = _indextime
| eval index_lag = abs(_indextime - _time)
| stats count as "Number of events", max(index_lag) as "Max", avg(index_lag) as "Avg", perc95(index_lag) as "95th Percentile", min(index_lag) as "Min" by host

 

Change the first line to match the index you are searching (wineventlog?  Windows?) and maybe add a sourcetype or whatever else. you need.

The second line - _indextime is hard to use "raw" because it's a special field.  So we just eval a new copy of it into a new field to use later.

The third line is the meat of calculating the difference between the two values.  We call it "index_lag"

The last line does your stats.   It gives you, by host, the number of events over whatever time period you have in the time picker, the largest index_lag for that host, the average index_lag, the 95th percentile of the lag for that host, and the minimum lag for it too.

That should

a) Give you a smoking gun

b) Also give you a lot more ammunition and knowledge about other things to search for in answers here if you need to do more.

Happy Splunking!

-Rich

0 Karma
Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...