Search for Alert Monitoring.

[AL3Z]

hello,

Could anyone assist me in creating a correlation search to detect triggered alerts across all searches. This will enable us to monitor counts and automatically notify us if any situation escalates beyond control.

Thanks

[gcusello]

If you want the list of triggered alerts please try this:

index=_audit action=alert_fired ss_app=* 
| eval ttl=expiration-now() 
| search ttl>0 
| convert ctime(trigger_time) 
| table trigger_time ss_name severity 
| rename trigger_time as "Alert Time" ss_name as "Alert Name" severity as "Severity"

if instead tu want the list of alerts in your environment, you could use:

|rest/servicesNS/-/-/saved/searches 
| search alert.track=1 
| fields title description search disabled triggered_alert_count actions action.script.filename alert.severity cron_schedule

Ciao.

Giuseppe

[AL3Z]

@gcusello ,

How we can set the threshold for the second search like if any of the CS alerts more than 10 times it should trigger a notables !

[gcusello]

Hi @AL3Z ,

the second search only lists the alerts not the triggered ones.

If you want the triggered alerts you have to use the first.

If you want to use a threshold, please try this:

index=_audit action=alert_fired ss_app=* 
| eval ttl=expiration-now() 
| search ttl>0 
| convert ctime(trigger_time) 
| stats count BY ss_name severity 
| where count>10

If you're using Enterprise Security, you don't need to use a Correlation Search like this, but you could use the Risk Score for assets and identities, but it's too long to describe.

Ciao.

Giuseppe

[AL3Z]

Can we use this CS in ES ?
Could you pls guide me how we could use the Risk Score for assets and identities?

[gcusello]

Hi @AL3Z,

yes you can use it in cs, but you can also use Notables.

Anyway, as action when an alert is triggered, you can define a Risk Score to assign to an asset or to an identity instead to trigger an alert.

Then you can define a threshold for the risk score, so, you'll have a Notable when the risk score, for an asset or an identity exceeds the threshold.

See in the Actions from a Correlation Search the Risk Score and make some try, I cannot guide you more.

For more infos see at https://docs.splunk.com/Documentation/ES/7.2.0/RBA/Analyzerisk

Ciao.

Giuseppe

[AL3Z]

@gcusello ,

Why we are not seeing the alerts for the disabled CS using the above search ?

[gcusello]

Hi @AL3Z ,

the above search lists the triggered alerts, if an alert is disablen is also never triggered|

Ciao.

Giuseppe

[AL3Z]

@gcusello ,

Why the triggered alerts from the search  are not matching with the incident review alerts why so ?

[gcusello]

Hi @AL3Z,

they match with the correlation searches, but thei contain also other alerts outside ES and anyway don't matcj with disabled CS.

Ciao.

Giuseppe

[AL3Z]

@gcusello 

I had chosen to specify only ES and I haven't consider the disabled CS  but still, not all the alerts are showing up.

[gcusello]

Hi @AL3Z,

I don't know: the above search lists all the triggered alerts.

Otherwise you could run a search on the notable index and have the count for the triggered searches:

index=notable
| stats count BY search_name
| where count>10

Ciao.

Giuseppe

[AL3Z]

@gcusello ,

How to configure this search as a alert scheduling?

threshold should be  2 seconds....

Thanks

[gcusello]

Hi @AL3Z,

run tis search and click on Save as.

Ciao.

Giuseppe

[AL3Z]

@gcusello ,

how we can complete this mark a threshold, 
alert count for all my searches alerts if the count >10 for last 7 days reads  (189,186,167,167,89,74,60,59,56,46,35,32,28,26,20,19,17,14,11).
How often do we need to run this in a day?

[gcusello]

Hi @AL3Z,

the run frequency depends on what's the max delay is acceptable for your in discovering the triggered alert: one day, one hour, I don't know, it depends by your requisites.

Ciao.

Giuseppe

[AL3Z]

@gcusello 

Which one would be better running it daily or realtime can you pls suggest we are into security specific usecases

[gcusello]

Hi @AL3Z,

avoid to use realtime!

what's your requirement: one day, use one day, there isn't a reccomandation, it's only related to your requirement.

Ciao.

Giuseppe

[AL3Z]

Hi @gcusello ,

Can we use it as a CS to trigger an alert when it exceeds the alert threshold ?

[gcusello]

Hi @AL3Z,

if you don't use the original CS, you don't have the notables for searching.

For this reason I hinted to use the risk score instead the notable as action.

You need someone that really knows ES to guide you or a training on ES using or administering.

Ciao.

Giuseppe

[AL3Z]

@gcusello 

I didn't get what is original CS mean is that with index notable or previous search ?
can you pls guide me or share me link to get master in ES!