Generete event after 3 consecutive failure

[nanoo1]

Hi,

I need an help with splunk search query where in an incident need to be generated for a log backup failure after 3 consecutive failures. 

/nanoo1

[nanoo1]

My query currently generating an incident for each failure.

For "log backup",  I would like an incident to generate after 3 failures, something log backup failure >3 , create an incident.

Impact=case(
message like("%Failed log backup of Oracle Database%"),"2-Significant/Large",
message like("%Failed backup of Oracle Database%"),"2-Significant/Large",
true(), "3-Moderate/Limited"),
Urgency=case(
message like("%Failed log backup of Oracle Database%"), "2-High",
message like("%Failed backup of Oracle Database%"), "2-High",
true(), "3-Medium" )

Below are the fields from my table table _time,objectName,message,locationName,eventStatus,objectType,objectId

here objectName related database name and objectType as Oracledb

[ITWhisperer]

| stats count by Urgency
| where count > 3

[nanoo1]

Thanks but we have other failures as well in this query and this field should be applicable for only Oracle log backups. Any suggestions how to embed this ?

[nanoo1]

I tried in this fashion

itsi_message="Alerting time: "+human_readable_time+"~~"+field1+"~~"+field2+"~~"+field3+"~~"+field4+"~~"+field5+"~~"+field6+"~~"+field7+"~~"+field8,
itsi_impact=case(
message like("%Failed log backup of Oracle Database%") ,"High" | stats count by "%Failed log backup of Oracle Database%" | where count >3,
message like("%Failed backup of Oracle Database%"),"High",
true(), "Medium") 

getting an eval error.

[ITWhisperer]

Yes, that's not going to work like that. Perhaps if you provided your full search we might be able to adjust it to give you the count you want

[nanoo1]

Here is my complete search:

| from datamodel:"dataset_backup_job_events"| where clusterName=="ITSEDC07-SD02A" | where eventStatus=="Failure" | table _time,objectName,message,locationName,eventStatus,objectType,objectId,_raw
| eval json_field=split(_raw,",")
| eval field1=mvindex(json_field,1)
| eval field1=replace(field1,"\"","")
| eval field2=mvindex(json_field,2)
| eval field2=replace(field2,"\"","")
| eval field3=mvindex(json_field,3)
| eval field3=replace(field3,"\"","")
| eval field4=mvindex(json_field,4)
| eval field4=replace(field4,"\"","")
| eval field5=mvindex(json_field,5)
| eval field5=replace(field5,"\"","")
| eval field6=mvindex(json_field,6)
| eval field6=replace(field6,"\"","")
| eval field7=mvindex(json_field,7)
| eval field7=replace(field7,"\"","")
| eval field8=mvindex(json_field,8)
| eval field8=replace(field8,"\"","")
| eval field8=rtrim(field8,"}")
| eval human_readable_time=strftime(_time, "%Y-%d-%m %H:%M")
| eval
itsi_entity=objectName,
itsi_event_key=objectId,
itsi_correlation_key=objectId,
itsi_summary="Backup "+eventStatus+" for "+objectName,
message=message,
itsi_tag=mvappend("NowIT", "ITSI"),
itsi_message="Alerting time: "+human_readable_time+"~~"+field1+"~~"+field2+"~~"+field3+"~~"+field4+"~~"+field5+"~~"+field6+"~~"+field7+"~~"+field8,
itsi_impact=case(
message like("%Failed log backup of Oracle Database%") ,"High",
message like("%Failed backup of Oracle Database%"),"High",
true(), "Medium"),
itsi_urgency=case(
message like("%Failed log backup of Oracle Database%"), "High",
message like("%Failed backup of Oracle Database%"), "High",
true(), "Medium")
| rex mode=sed field=itsi_message "s/\\\/-/g"

[ITWhisperer]

How about something like this

| stats count(eval(message like("%Failed log backup of Oracle Database%"))) values(message) as messages by objectName
| where count > 3

[nanoo1]

I tried to add these lines 

itsi_impact=case(
| stats count(eval(message like("%Failed log backup of Oracle Database%"))) values(message) as messages by objectName,
| where count > 3,
message like("%Failed log backup of Oracle Database%") ,"High",
message like("%Failed backup of Oracle Database%"),"High",
true(), "Medium")

Getting an error "Error in 'eval' command: The expression is malformed. Expected ) "

I tried to keep outside as well but eval is expecting= , any suggestion ?

[ITWhisperer]

Try adding after your final rex

[nanoo1]

Hi,

I added after final regex but no luck

[ITWhisperer]

Please provide examples of the log events you are working with

[nanoo1]

One of the log event example  

time: 2021-12-15T15:54:17.725Z
   clusterName: ITSEDC08-SD02D
   eventStatus: Failure
   eventType: Backup
   id: f78ad415-6fde-49aa-b004-58cde954e783
   message: Failed log backup of Oracle Database 'ITC1089'.
   objectId: OracleDatabase:::94dea975-e1fe-4684-8b7e-9e77e4b92e81
   objectName: ITC1089
   objectType: OracleDb

[nanoo1]

Hi,

Any other suggestions ?