Splunk alert based on custom search condition

[worldexplorer81]

I have multiple jobs that run throughout the day and they complete at different times with statusText of FAILURE, SUCCESS, TERMINATED

I need to create an alert that would sent an email for all jobNames that failed in the last 12 hours, with the timestamp. However, the alert should only be triggered once, upon the completion of a specific jobname (i.e once jobName test_job100 has a status of FAILURE or SUCCESS)

All jobNames start with: “test_%”
The specific jobname which should trigger the alert also starts with test_: test_job100

Below is an example of the log

2020-05-01 12:11:01.194, timestamp="2020-05-01 12:09:57.0", jobId="280568", jobName="test_job6", boxJobName=" ", eventCode="101", eventText="CHANGE_STATUS", statusCode="5", statusText="FAILURE", alarmCode="0", exitCode="1"

Can someone advise how can I achieve this?

[shivanshu1593]

Not a lot of work with, but you can try something like this. I'm assuming the data for all the jobs is stored inside the same Index, I'll call it foo.

index=foo sourcetype=bar earliest=-12h statusText="FAILURE" | table jobName, _time

When you save this as an alert, in the Trigger condition section, click on the drop down and select custom and try and specify your condition like jobName=test_job100 AND (statusText="FAILURE" OR statusText="SUCCESS")

This will make sure that the alert is fired when your test_job100 ran and gave you a status of success or failure.

Cheers,

[worldexplorer81]

Thanks - I'll give it a try

[worldexplorer81]

unfortunately that did not work. I added the below to the custom trigger condition but nothing was triggered
search jobName=test_job100 AND (statusText="FAILURE" OR statusText="SUCCESS")

[shivanshu1593]

Try the following:

index=foo sourcetype=bar jobName="*" OR (jobName=test_job100 AND statusText="FAILURE" OR statusText="SUCCESS") earliest=-12h latest=now() | table jobName, _time | where statusText="FAILURE".

Set thid query as an alert and let me know if it works.

Ciao,

[shivanshu1593]

When setting the query that I just posted, please do not use the custom alert condition. I've tried to to fit it in inside the SPL itself.