Reduce # of Repeated Alerts

[Ewong]

Hi all,

I have the following Correlation Search set up to detect accounts that have been excessively locked out during a short period of time. However, we get many repeat alerts even if the last lockout time is long past (probably because of how it is configured). Is there a way to stop the alerting once the lockouts stop?

For example, an account has been locked out an excessive number of times (ie. 10 times); the first time this occurred is on 05/01/2021 at 09:34:03 and the last time is 05/01/2021 at 18:22:08, we would still get alerted days after the fact (ie. 05/05/2021).

Correlation Search:

Mode: Manual
Search:

index=wineventlog EventCode=4740
| stats count min(_time) as firstTime max(_time) as lastTime values(dest_nt_domain) as machines
by user, signature
| `ctime(firstTime)` | `ctime(lastTime)`
| search count > 5

Time Range:

Earliest Time: -7d

Latest Time: -10m@m

Cron Schedule: */15****

Scheduling: Continuous

Schedule Window: Auto

Schedule Priority: Default

Trigger Conditions:
Trigger alert when: Number of Results is greater than 0

Throttling:

Window duration: 1 day(s)

Fields to group by: user

[tscroggins]

@Ewong 

You're currently looking for five or more occurrences of EventCode=4740 every 15 minutes over the prior seven days. E.g. If four events occurred six days and one event occurred today, you would generate a notable event. Is this behavior you see? The solution is to decrease your search time range.

Also note that EventCode=4740 is not necessarily unique to Windows security events. Generally, you would uniquely identify Windows event log events by event log, registered event source, and event identifier, although third-party software often fails to adhere to Microsoft's event logging guidelines. This 3-tuple allows Windows to localize event log messages by ordinal (the event identifier) in a locale specific resource DLL for the registered event source. The message you see in Splunk was rendered by Windows in the server's default locale. (And I'll stop here before it turns into a Windows history lesson! 🙂

[Ewong]

Correct, that's the expected behavior. I was wondering if decreasing the search time range would omit or cause any events uncaptured. 

And I appreciate the Windows history lesson! 

[tscroggins]

@Ewong 

It may, but right now, you're asking Splunk for any six lockouts over the last seven days. You should shorten your time range to your window of interest relative to your backend failed login threshold, attempt backoff delay, indexing lag, etc.

The evolution of MS-DOS and Windows is fascinating!