topic How to trigger an action when no results are found (without using the settings in a saved alert) in Splunk Search

How to trigger an action when no results are found (without using the settings in a saved alert)

lordhans — Tue, 29 Sep 2020 16:46:55 GMT

My organization using something called Ticketer to in Splunk to auto-generate an incident form when something shows up in the logs. An example would look like this:

ns="my_application_namespace" "Exception X has occurred" | eval severity="4" | eval emailAddress="myTeam@somewhere.com" | eval description = "Exception X has occured" | ticketer

I saved that alert and it runs once an hour and if "Exception X has occurred" shows up in the logs anywhere an incident is created. Note that this isn't configured in the saved alert, it is part of the search query.

Now, what want to do is run a search query for a healthcheck url to make sure a service is up and running. If I don't see the healthcheck url appearing in the logs for an hour it means the service is down and an incident needs to be created.

This is what I have so far:
ns="my_application_namespace" "healthcheckUrlHere" | eval severity="4" | eval emailAddress="myTeam@somewhere.com" | eval description = "The Servive is down" | ticketer

I'm thinking I could use something like | stats count | search count=0

Or is there a better way of doing something like this that I don't know about?

Thanks for any help!

Re: How to trigger an action when no results are found (without using the settings in a saved alert)

micahkemp — Tue, 14 Nov 2017 00:57:17 GMT

If you're only looking for a single healthCheckUrl this is likely pretty simple, and what you have above should be fine.

If you wanted to abstract it out to multiple ns or healthCheckUrl values it would be more involved.

Re: How to trigger an action when no results are found (without using the settings in a saved alert)

hardikJsheth — Tue, 14 Nov 2017 05:37:52 GMT

Your query looks good. You can combine different eval statement in single command.

ns="my_application_namespace" "healthcheckUrlHere" | stats count | search count=0 | eval severity="4", emailAddress="myTeam@somewhere.com" , description = "The Servive is down" | ticketer

Re: How to trigger an action when no results are found (without using the settings in a saved alert)

gcusello — Tue, 14 Nov 2017 07:58:13 GMT

Hi lordhans,
knowing the list of processes to healthcheck I'd think to a different approach creating a lookup containing the processes to check:

Your_search [ | inputlookup processes.csv | fields process ]
| eval process=lower(process)
| stats count by process
| append [ | inputlookup processes.csv | eval process=lower(process), count=0 | fields process count ]
| stats sum(count) AS Total BY process

In this way, processes where Total=0 are missed and processes where Total>0 are OK.

Bye.
Giuseppe