Alerting

Splunk alert based on custom search condition

worldexplorer81
Path Finder

I have multiple jobs that run throughout the day and they complete at different times with statusText of FAILURE, SUCCESS, TERMINATED

I need to create an alert that would sent an email for all jobNames that failed in the last 12 hours, with the timestamp. However, the alert should only be triggered once, upon the completion of a specific jobname (i.e once jobName test_job100 has a status of FAILURE or SUCCESS)

All jobNames start with: “test_%”
The specific jobname which should trigger the alert also starts with test_: test_job100

Below is an example of the log

2020-05-01 12:11:01.194, timestamp="2020-05-01 12:09:57.0", jobId="280568", jobName="test_job6", boxJobName=" ", eventCode="101", eventText="CHANGE_STATUS", statusCode="5", statusText="FAILURE", alarmCode="0", exitCode="1"

Can someone advise how can I achieve this?

Labels (3)
Tags (2)
0 Karma

shivanshu1593
Builder

Not a lot of work with, but you can try something like this. I'm assuming the data for all the jobs is stored inside the same Index, I'll call it foo.

index=foo sourcetype=bar earliest=-12h statusText="FAILURE" | table jobName, _time

When you save this as an alert, in the Trigger condition section, click on the drop down and select custom and try and specify your condition like jobName=test_job100 AND (statusText="FAILURE" OR statusText="SUCCESS")

This will make sure that the alert is fired when your test_job100 ran and gave you a status of success or failure.

Cheers,

Thank you,
Shiv
###If you found the answer helpful, kindly consider upvoting/accepting it as the answer as it helps other Splunkers find the solutions to similar issues###
0 Karma

worldexplorer81
Path Finder

Thanks - I'll give it a try

0 Karma

worldexplorer81
Path Finder

unfortunately that did not work. I added the below to the custom trigger condition but nothing was triggered
search jobName=test_job100 AND (statusText="FAILURE" OR statusText="SUCCESS")

0 Karma

shivanshu1593
Builder

Try the following:

index=foo sourcetype=bar jobName="*" OR (jobName=test_job100 AND statusText="FAILURE" OR statusText="SUCCESS") earliest=-12h latest=now() | table jobName, _time | where statusText="FAILURE".

Set thid query as an alert and let me know if it works.

Ciao,

Thank you,
Shiv
###If you found the answer helpful, kindly consider upvoting/accepting it as the answer as it helps other Splunkers find the solutions to similar issues###
0 Karma

shivanshu1593
Builder

When setting the query that I just posted, please do not use the custom alert condition. I've tried to to fit it in inside the SPL itself.

Thank you,
Shiv
###If you found the answer helpful, kindly consider upvoting/accepting it as the answer as it helps other Splunkers find the solutions to similar issues###
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...