Splunk Search

How to trigger an action when no results are found (without using the settings in a saved alert)

lordhans
Explorer

My organization using something called Ticketer to in Splunk to auto-generate an incident form when something shows up in the logs. An example would look like this:

ns="my_application_namespace" "Exception X has occurred" | eval severity="4" | eval emailAddress="myTeam@somewhere.com" | eval description = "Exception X has occured" | ticketer

I saved that alert and it runs once an hour and if "Exception X has occurred" shows up in the logs anywhere an incident is created. Note that this isn't configured in the saved alert, it is part of the search query.

Now, what want to do is run a search query for a healthcheck url to make sure a service is up and running. If I don't see the healthcheck url appearing in the logs for an hour it means the service is down and an incident needs to be created.

This is what I have so far:
ns="my_application_namespace" "healthcheckUrlHere" | eval severity="4" | eval emailAddress="myTeam@somewhere.com" | eval description = "The Servive is down" | ticketer

I'm thinking I could use something like | stats count | search count=0

Would this work if I inserted it before the eval portion of the search? Like: ns="my_application_namespace" "healthcheckUrlHere" | stats count | search count=0 | eval severity="4" | eval emailAddress="myTeam@somewhere.com" | eval description = "The Servive is down" | ticketer

Or is there a better way of doing something like this that I don't know about?

Thanks for any help!

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi lordhans,
knowing the list of processes to healthcheck I'd think to a different approach creating a lookup containing the processes to check:

Your_search [ | inputlookup processes.csv | fields process ]
| eval process=lower(process)
| stats count by process
| append [ | inputlookup processes.csv | eval process=lower(process), count=0 | fields process count ]
| stats sum(count) AS Total BY process

In this way, processes where Total=0 are missed and processes where Total>0 are OK.

Bye.
Giuseppe

0 Karma

hardikJsheth
Motivator

Your query looks good. You can combine different eval statement in single command.

ns="my_application_namespace" "healthcheckUrlHere" | stats count | search count=0 | eval severity="4", emailAddress="myTeam@somewhere.com" , description = "The Servive is down" | ticketer
0 Karma

micahkemp
Champion

If you're only looking for a single healthCheckUrl this is likely pretty simple, and what you have above should be fine.

If you wanted to abstract it out to multiple ns or healthCheckUrl values it would be more involved.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...