Alerting

Prevent repeated alerts on recurring results

calejohn5
Explorer

Hello all,

I've recently been tasked with alerting our support email when a user in Salesforce is locked out.  The alert triggers when a User's LOGIN_STATUS="LOGIN_ERROR_PASSWORD_LOCKOUT".  However, this alert keeps getting triggered if an admin doesn't unlock the User account right away.  Is there any way to limit the alert being sent out if the Usernames are identical as the previous alert?

index="salesforce" EVENT_TYPE="Login" LOGIN_STATUS=*
[search EVENT_TYPE="Login" LOGIN_STATUS="LOGIN_ERROR_PASSWORD_LOCKOUT"
| stats count by USER_ID
| table USER_ID]
| stats latest(LOGIN_STATUS) AS LOGIN_STATUS
latest(USER_NAME) AS USER_NAME
latest(SOURCE_IP) AS SOURCE_IP
latest(UserAccountId) AS "Account Id"
latest(USER_TYPE) AS "User Type"
latest(TIMESTAMP) AS "Time stamp" by USER_ID
| where LOGIN_STATUS="LOGIN_ERROR_PASSWORD_LOCKOUT"

Labels (3)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Turn on throttling.  

richgalloway_0-1633047320650.png

 

---
If this reply helps you, Karma would be appreciated.

calejohn5
Explorer

When I add Throttling does that suppress alerts ONLY for that user?  Because I want it to continue to alert for different users getting locked out while that initial locked user is locked out

0 Karma

richgalloway
SplunkTrust
SplunkTrust

If the alert is triggered once then the throttle applies to all instance of that alert for the throttle period.

If the alert is triggered once per result then you can specify a throttle condition.  See https://community.splunk.com/t5/Alerting/Suppress-results-containing-field-value-with-multiple-value...

---
If this reply helps you, Karma would be appreciated.
0 Karma

calejohn5
Explorer

Ok I turned on For Each Result and Suppressing when USER_ID= $result.USER_ID$.  I just need to wait for this query to run again and get indexed and I'll see if that works. 

0 Karma

calejohn5
Explorer

It seemed to not send out any alerts...

Is there any way to index data from another instance faster?  The data doesn't seem updated when I search for event I know recently happened.  This would help with testing 10-fold.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I don't know why no alerts would be sent unless none were triggered.

Tell us more about your other request.  What do you mean by "index data faster"?  What is the other instance?  What is the data flow?

---
If this reply helps you, Karma would be appreciated.
0 Karma

calejohn5
Explorer

When I hit "Open in Search" for the alert it brings me to the search, except it's timeframe is "Last 1 hour" while the actual search I built this alert from is based off "All Time".

The "Last 1 hour" timeframe doesn't yield any results whereas the "All Time" search yields a result, which would trigger the alert.

So maybe this search already limits this return result to one time...

 

But in regards to my last request - Whenever I perform an event in Salesforce, it takes about 8 hours for that event to get added into Splunk.  This makes testing extremely difficult.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I think the "open in search" link uses the default time window for your system.  Manually adjust it to the desired window.

The problem with getting data from Salesforce warrants a new question.

---
If this reply helps you, Karma would be appreciated.

calejohn5
Explorer

Ok that makes sense and I tried researching why my data was taking so long to enter Splunk from Salesforce but gave up.  It's not a big deal as I just need to fix this one alert anyway.

I was thinking - If I chose number of results rises by '1' for the trigger condition, wouldn't that send out an individual alert every time a new event (user lockout) occurs, and never again for the same result since the count would remain the same?

I'm testing that out, but as I said earlier it may take awhile.

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I have not tried the "rises by 1" option so I don't know if it will help or not.  Please report your findings.

---
If this reply helps you, Karma would be appreciated.
0 Karma

calejohn5
Explorer

I ran out of time to properly test "rise by 1".  I couldn't even get $result.USER_ID$ in the alert message to work.

I ended up just triggering for each result and suppressing the alert for 24 hours.  If anything I'll have to come back in the future and learn more/work on this.  I appreciate the help!

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...