How to trigger an action when no results are found (without using the settings in a saved alert)

[lordhans]

My organization using something called Ticketer to in Splunk to auto-generate an incident form when something shows up in the logs. An example would look like this:

ns="my_application_namespace" "Exception X has occurred" | eval severity="4" | eval emailAddress="myTeam@somewhere.com" | eval description = "Exception X has occured" | ticketer

I saved that alert and it runs once an hour and if "Exception X has occurred" shows up in the logs anywhere an incident is created. Note that this isn't configured in the saved alert, it is part of the search query.

Now, what want to do is run a search query for a healthcheck url to make sure a service is up and running. If I don't see the healthcheck url appearing in the logs for an hour it means the service is down and an incident needs to be created.

This is what I have so far:
ns="my_application_namespace" "healthcheckUrlHere" | eval severity="4" | eval emailAddress="myTeam@somewhere.com" | eval description = "The Servive is down" | ticketer

I'm thinking I could use something like | stats count | search count=0

Would this work if I inserted it before the eval portion of the search? Like: ns="my_application_namespace" "healthcheckUrlHere" | stats count | search count=0 | eval severity="4" | eval emailAddress="myTeam@somewhere.com" | eval description = "The Servive is down" | ticketer

Or is there a better way of doing something like this that I don't know about?

Thanks for any help!

[gcusello]

Hi lordhans,
knowing the list of processes to healthcheck I'd think to a different approach creating a lookup containing the processes to check:

Your_search [ | inputlookup processes.csv | fields process ]
| eval process=lower(process)
| stats count by process
| append [ | inputlookup processes.csv | eval process=lower(process), count=0 | fields process count ]
| stats sum(count) AS Total BY process

In this way, processes where Total=0 are missed and processes where Total>0 are OK.

Bye.
Giuseppe

[hardikJsheth]

Your query looks good. You can combine different eval statement in single command.

ns="my_application_namespace" "healthcheckUrlHere" | stats count | search count=0 | eval severity="4", emailAddress="myTeam@somewhere.com" , description = "The Servive is down" | ticketer

[micahkemp]

If you're only looking for a single healthCheckUrl this is likely pretty simple, and what you have above should be fine.

If you wanted to abstract it out to multiple ns or healthCheckUrl values it would be more involved.