splunk alert raising multiple SNOW tickets that are not stopping until I disable the alert

[avi123]

when I run this search query in splunk search and reporting apps my output looks like this as mentioned below

Search query:

index="dcn_b2b_use_case_analytics" sourcetype=lime_process_monitoring

Output:

Time 

3/19/25 2:32:15.000 PM
Event
{ [-]

    BCD_AB_UY_01: 1

    BCD_AB_UY_02: 0

    BCD_BC_01: 1

    BCD_BC_02: 0

    BCD_CD_01: 1

    BCD_CD_02: 1

    BCD_CD_03: 0

    BCD_KPI_01: 1

    BCD_KPI_02: 1

    BCD_KPI_03: 0

    BCD_MY_01: 1

    BCD_MY_02: 1

    BCD_RMO_PZ_01: 1

    BCD_RMO_PZ_02: 1

    BCD_RMO_PZ_03: 0

    BCD_RMO_PZ_04: 0

    BCD_RSTA_01: 1

    BCD_RSTA_02: 1

    BCD_RSTA_03: 0

    BCD_SHY_01: 1

    BCD_SHY_02: 1

    BCD_UK_01: 1

    BCD_UK_02: 1

    BCD_UK_03: 1

    BCD_UK_04: 1

    BCD_UK_05: 1

    BCD_UK_06: 1

    BCD_UK_07: 1

    BCD_UK_08: 0

    BCD_UK_09: 0

    BCD_UK_10: 0

    BCD_UK_11: 0

    BCD_UK_12: 0

}

host = RSQWERTYASD04index = dcn_b2b_use_case_analyticssource = DCNPassFoldersourcetype = lime_process_monitoring

Please Note- if a process value is 1 it means the process ran successfully, if it is 0 it means the process failed

Now my query is I want to trigger an alert for these processes mentioned below so that when these background processes fail I get an incident in my queue 

in SNOW

BCD_AB_UY_01: 0

BCD_BC_01: 0

BCD_CD_01: 0

BCD_CD_02: 0

BCD_KPI_01: 0

BCD_KPI_02: 0

BCD_MY_01: 0

BCD_MY_02: 0

BCD_RMO_PZ_01: 0

BCD_RMO_PZ_02: 0

BCD_RSTA_01: 0

BCD_RSTA_02: 0

BCD_SHY_01: 0

BCD_SHY_02: 0

BCD_UK_01: 0

BCD_UK_02: 0

BCD_UK_03: 0

BCD_UK_04: 0

BCD_UK_05: 0

BCD_UK_06: 0

BCD_UK_07: 0

This is the alert search query I designed but when I run this alert I get multiple tickets instead I want a particular ticket where servicename(process name)

and servername(hostname) is clearly mentioned to uniquely identify the process is from which server, please help me write and configure the splunk alert properly:

Search query-

index="dcn_b2b_use_case_analytics" sourcetype=lime_process_monitoring 

| where BGS_AR_UY_01=0 OR BGS_BR_01=0 OR BGS_BS_01=0 OR BGS_BS_02=0 OR BGS_KAU_01=0 OR BGS_KAU_02=0 OR BGS_MX_01=0 

OR BGS_MX_02=0 OR BGS_RMH_PZ_01=0 

OR BGS_RMH_PZ_02=0 OR BGS_RSTO_01=0 OR BGS_RSTO_02=0 OR BGS_SHA_01=0 OR BGS_SHA_02=0 

OR BGS_US_01=0 OR BGS_US_02=0 OR BGS_US_03=0 OR BGS_US_04=0 OR 

BGS_US_05=0 OR BGS_US_06=0 OR BGS_US_07=0   

| eval metricLabel="URGENT !! Labware - < ServiceName > has been stopped in Server"

| eval metricValue="Hello Application Support team,

The below service has been stopped in the server,

Service name :  < ServiceName > Timestamp :  < Timestamp > 

 Server : <ServerName> 

Please take the required action to resume the service. Thank you.

Regards,

Background Service Check Automation Bot"

| eval querypattern="default"

| eval assignmentgroup="PTO ABC Lab - Operatives"

| eval business_service="LIME Business Service"

| eval serviceoffering="LIME"

| eval Interface="CLMTS"

| eval urgency=2

| eval impact=1

Cron expression * * * * *

Trigger For each result

Trigger actions PTIX SNOWALERT

[PickleRick]

Wait a second.

You have two separate issues here.

One is your search - whether it can be written better (yes, it can) is one question. But another question is how the search is run - how often it is spawned (in your case it's every minute; isn't it a bit too much?), over what time range it is being run and thus how many results it returns and also what is your data ingestion characteristics - how often you get new events?

For example, if you're searching every minute over last 15 minutes worth of data, you will hit the same result for about 15 times (the actual searches of course might get delayed or skipped depending on your search load and the schedule type) so unless you use throttling, you'll get 15 separate alerts.

[livehybrid]

Hi @avi123 

Its getting late here so might not be optimal, but I think this should work!

| makeresults 
| eval json_data="{\"BCD_AB_UY_01\":1,\"BCD_AB_UY_02\":0,\"BCD_BC_01\":1,\"BCD_BC_02\":0,\"BCD_CD_01\":1,\"BCD_CD_02\":1,\"BCD_CD_03\":0,\"BCD_KPI_01\":1,\"BCD_KPI_02\":1,\"BCD_KPI_03\":0,\"BCD_MY_01\":1,\"BCD_MY_02\":1,\"BCD_RMO_PZ_01\":1,\"BCD_RMO_PZ_02\":1,\"BCD_RMO_PZ_03\":0,\"BCD_RMO_PZ_04\":0,\"BCD_RSTA_01\":1,\"BCD_RSTA_02\":1,\"BCD_RSTA_03\":0,\"BCD_SHY_01\":1,\"BCD_SHY_02\":1,\"BCD_UK_01\":1,\"BCD_UK_02\":1,\"BCD_UK_03\":1,\"BCD_UK_04\":1,\"BCD_UK_05\":1,\"BCD_UK_06\":1,\"BCD_UK_07\":1,\"BCD_UK_08\":0,\"BCD_UK_09\":0,\"BCD_UK_10\":0,\"BCD_UK_11\":0,\"BCD_UK_12\":0}" 
| eval _raw=json_extract(json_data,"") 
| eval host="Testing", service="MySerivceName" 
| spath 
| foreach * 
    [| eval fields=mvappend(fields, IF(<<FIELD>> >= 0, json_object("<<FIELD>>",<<FIELD>>),null()))] 
| table _time host service fields 
| mvexpand fields 
| eval fieldObj=json_array_to_mv((json_entries(fields))) 
| eval fieldName=json_extract(fieldObj, "key")
| eval value=json_extract(fieldObj, "value")
| eval friendlyTime=strftime(_time,"%d/%m/%Y %H:%M:%S")
| search value=0 
| eval metricLabel="URGENT !! Labware - ".service." has been stopped in Server" 
| eval metricValue="Hello Application Support team,
The below service has been stopped in the server,
Service name :  ".service." Timestamp :  ".friendlyTime."
 Server : ".host."
Please take the required action to resume the service. Thank you.
Regards,
Background Service Check Automation Bot" 
| eval querypattern="default" 
| eval assignmentgroup="PTO ABC Lab - Operatives" 
| eval business_service="LIME Business Service" 
| eval serviceoffering="LIME" 
| eval Interface="CLMTS" 
| eval urgency=2 
| eval impact=1

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will