Alerting
Highlighted

How to create an alert to trigger when an index is close to meeting its retention requirement before running out of space?

Builder

All,

What I am trying to do now is get an alert when an index is close to making its retention requirement before running out of space. Anyone have a search they can share?

thank in advance!
-Daniel

0 Karma
Highlighted

Re: How to create an alert to trigger when an index is close to meeting its retention requirement before running out of space?

SplunkTrust
SplunkTrust

Would this be about the same as wanting a search that compares the two percentages A and B, where

A = the percentage filled an index is, which is consumed space / maximum space
B = the percentage of how much of the expected retention is in the index, which is more complex and should be something like oldest event available / how old the max retention is.

So if you are 50% filled but at only 45% of your retention, you'd want an alert? (Probably doing to run out of space before you hit your retention)

And if you were 50% filled but at 60% of your retention, you're OK? (You have enough space to hold your retention settings, at least probably)

Right?

0 Karma
Highlighted

Re: How to create an alert to trigger when an index is close to meeting its retention requirement before running out of space?

Esteemed Legend

Create a lookup called index_retention_goal with fields indexname and retentionGoalDays, and thresholdPct. Then run this as an alert:

index=_internal sourcetype=splunkd bucketmover "will attempt to freeze" | rex field=_raw "/splunkdata(?:/[^/]*)?/(?<indexname>[^/]*)/db/db_(?<newestTime>[^_]*)_(?<oldestTime>[^_]*)_.*" | dedup indexname | eval retentionDays=(now()-oldestTime)/(60*60*24) | stats values(retentionDays) as retentionDays by indexname | lookup index_retention_goal indexname | eval currentPct=((100 * retentionDays) / retentionGoalDays) | where currentPct >= thresholdPct
0 Karma