How to create an alert to trigger when an index is close to meeting its retention requirement before running out of space?

[daniel333]

All,

What I am trying to do now is get an alert when an index is close to making its retention requirement before running out of space. Anyone have a search they can share?

thank in advance!
-Daniel

[woodcock]

Create a lookup called index_retention_goal with fields indexname and retentionGoalDays, and thresholdPct. Then run this as an alert:

index=_internal sourcetype=splunkd bucketmover "will attempt to freeze" | rex field=_raw "/splunkdata(?:/[^/]*)?/(?<indexname>[^/]*)/db/db_(?<newestTime>[^_]*)_(?<oldestTime>[^_]*)_.*" | dedup indexname | eval retentionDays=(now()-oldestTime)/(60*60*24) | stats values(retentionDays) as retentionDays by indexname | lookup index_retention_goal indexname | eval currentPct=((100 * retentionDays) / retentionGoalDays) | where currentPct >= thresholdPct

[Richfez]

Would this be about the same as wanting a search that compares the two percentages A and B, where

A = the percentage filled an index is, which is consumed space / maximum space
B = the percentage of how much of the expected retention is in the index, which is more complex and should be something like oldest event available / how old the max retention is.

So if you are 50% filled but at only 45% of your retention, you'd want an alert? (Probably doing to run out of space before you hit your retention)

And if you were 50% filled but at 60% of your retention, you're OK? (You have enough space to hold your retention settings, at least probably)

Right?