How to Throttle a Scheduled Alert by Field?

[rackersmt]

Splunk receives logs from our domain controllers, and we're wanting to trigger on accounts that have an excessive number of lockout events (event code 4740). A real-time alert causes too much load on the search head because of all the 4740 events coming in (we're a very large organization). Therefore, I've switched to a Scheduled Alert that runs hourly with an action of sending an email. I'd like to not send the email if the account that's locked out has been alerted on in the last 24 hours. Since Scheduled Alerts can't seem to throttle by field, is there a way to alert only if new domain accounts how up in the report?

Here's the basic search I'm running for the alert without throttling:

index=[index-name] EventCode=4740 earliest=-24h | rex field=_raw "(?s)(?i)Account\sThat\sWas\sLocked\sOut.+?Account\sName:\s+(?\S*)" | search NOT acct_name="Guest" | top acct_name | where count >= 50 | sort -count

Thanks!

[somesoni2 · Accepted solution]

Try something like this.

index=[index-name] EventCode=4740 earliest=-25h | rex field=_raw "(?s)(?i)Account\sThat\sWas\sLocked\sOut.+?Account\sName:\s+(?\S*)" | search NOT acct_name="Guest" | eval Period=if(_time>relative_time(now(),"-1h"),"New","Old" ) | stats count values(Period) as Period by acct_name  | where mvcount(Period)=1 AND  Period="New" AND count >= 50| sort -count | head 10 | fields -Period

SInce, you're running the search every hour, the new lockouts will be there in last 1 hour period only, others would've been alerted during previous hour. So, above search check if the lockout has only happened in last 1 hr (period=New) with count>50.

[rackersmt]

Beautiful, thank you!