Prevent repeated alerts on recurring results

[calejohn5]

Hello all,

I've recently been tasked with alerting our support email when a user in Salesforce is locked out.  The alert triggers when a User's LOGIN_STATUS="LOGIN_ERROR_PASSWORD_LOCKOUT".  However, this alert keeps getting triggered if an admin doesn't unlock the User account right away.  Is there any way to limit the alert being sent out if the Usernames are identical as the previous alert?

index="salesforce" EVENT_TYPE="Login" LOGIN_STATUS=*
[search EVENT_TYPE="Login" LOGIN_STATUS="LOGIN_ERROR_PASSWORD_LOCKOUT"
| stats count by USER_ID
| table USER_ID]
| stats latest(LOGIN_STATUS) AS LOGIN_STATUS
latest(USER_NAME) AS USER_NAME
latest(SOURCE_IP) AS SOURCE_IP
latest(UserAccountId) AS "Account Id"
latest(USER_TYPE) AS "User Type"
latest(TIMESTAMP) AS "Time stamp" by USER_ID
| where LOGIN_STATUS="LOGIN_ERROR_PASSWORD_LOCKOUT"

[richgalloway]

Turn on throttling.  

[calejohn5]

When I add Throttling does that suppress alerts ONLY for that user?  Because I want it to continue to alert for different users getting locked out while that initial locked user is locked out

[richgalloway]

If the alert is triggered once then the throttle applies to all instance of that alert for the throttle period.

If the alert is triggered once per result then you can specify a throttle condition.  See https://community.splunk.com/t5/Alerting/Suppress-results-containing-field-value-with-multiple-value...

[calejohn5]

Ok I turned on For Each Result and Suppressing when USER_ID= $result.USER_ID$.  I just need to wait for this query to run again and get indexed and I'll see if that works. 

[calejohn5]

It seemed to not send out any alerts...

Is there any way to index data from another instance faster?  The data doesn't seem updated when I search for event I know recently happened.  This would help with testing 10-fold.

[richgalloway]

I don't know why no alerts would be sent unless none were triggered.

Tell us more about your other request.  What do you mean by "index data faster"?  What is the other instance?  What is the data flow?

[calejohn5]

When I hit "Open in Search" for the alert it brings me to the search, except it's timeframe is "Last 1 hour" while the actual search I built this alert from is based off "All Time".

The "Last 1 hour" timeframe doesn't yield any results whereas the "All Time" search yields a result, which would trigger the alert.

So maybe this search already limits this return result to one time...

But in regards to my last request - Whenever I perform an event in Salesforce, it takes about 8 hours for that event to get added into Splunk.  This makes testing extremely difficult.

[richgalloway]

I think the "open in search" link uses the default time window for your system.  Manually adjust it to the desired window.

The problem with getting data from Salesforce warrants a new question.

[calejohn5]

Ok that makes sense and I tried researching why my data was taking so long to enter Splunk from Salesforce but gave up.  It's not a big deal as I just need to fix this one alert anyway.

I was thinking - If I chose number of results rises by '1' for the trigger condition, wouldn't that send out an individual alert every time a new event (user lockout) occurs, and never again for the same result since the count would remain the same?

I'm testing that out, but as I said earlier it may take awhile.

[richgalloway]

I have not tried the "rises by 1" option so I don't know if it will help or not.  Please report your findings.

[calejohn5]

I ran out of time to properly test "rise by 1".  I couldn't even get $result.USER_ID$ in the alert message to work.

I ended up just triggering for each result and suppressing the alert for 24 hours.  If anything I'll have to come back in the future and learn more/work on this.  I appreciate the help!