topic Re: How to build an alert based on status code? in Splunk Search

How to build an alert based on status code?

vemurisurya — Mon, 08 Jun 2020 17:57:16 GMT

Hi,
i have 10 stats codes from 200 to 210, i need to set up an alert. That alert will look at the last 10 mins, if a stats code was not generated in last 10 min, Splunk should send an alert. How could i build a search for this?

Re: How to build an alert based on status code?

cpetterborg — Tue, 08 Aug 2017 16:33:47 GMT

Figuring you are using the GUI to enter the search, use -10m@m to now in the time picker.

Then use something like this for the search:

... stats_code>199 stats_code<211

Then when you create the alert, set the alert condition to alert if the count of events = 0.

Re: How to build an alert based on status code?

woodcock — Tue, 08 Aug 2017 17:39:40 GMT

Like this:

index="YouShouldAlwaysSpecifyAnIndex" sourcetype="AndSourcetypeToo" status_code>=200 AND status_code<=210 
| stats count
| where count=0

Schedule this to run every 5 minutes over the last 10 minutes and set to alert when number of events and is greater than 0

Re: How to build an alert based on status code?

cpetterborg — Tue, 08 Aug 2017 17:46:16 GMT

I think he wants the case where there are no events. He stated in the question:

if a stats code was not generated in last 10 min

Re: How to build an alert based on status code?

vemurisurya — Tue, 08 Aug 2017 17:58:24 GMT

yeah correct if no status code were generated in 10 min there splunk need to triggered an alert

Re: How to build an alert based on status code?

woodcock — Tue, 08 Aug 2017 19:08:35 GMT

That's what it does (the logic is in the search; I ALWAYS put my logic in the search and use when number of events and is greater than 0).

Re: How to build an alert based on status code?

cpetterborg — Tue, 08 Aug 2017 19:28:45 GMT

Oops, yeah. I've got egg on my face. 🙂 I didn't realize that was the result until you pointed that out to me. I think the explanation helps the understanding of your method. Thanks!

Re: How to build an alert based on status code?

vemurisurya — Wed, 09 Aug 2017 12:35:52 GMT

I need the alert report like, if any code not generate any code with in the last hour that should be send alert like
Status_code count
200 0
201 1
202 5
203 0

like this if any stats count was 0 then alert will triggered

Re: How to build an alert based on status code?

woodcock — Wed, 09 Aug 2017 13:47:54 GMT

That is different and not the plain reading of what you asked. See my new answer (soon).

Re: How to build an alert based on status code?

woodcock — Wed, 09 Aug 2017 13:48:54 GMT

Like this:

index="YouShouldAlwaysSpecifyAnIndex" sourcetype="AndSourcetypeToo" status_code>=200 AND status_code<=210 
| stats count BY status_code
| where count=0

Schedule this to run every 5 minutes over the last 10 minutes and set to alert when number of events and is greater than 0.

Re: How to build an alert based on status code?

vemurisurya — Wed, 09 Aug 2017 14:54:54 GMT

this will shows only status code exist in the events , if no stats code in the events , it doesn't shows any value, so here no visibility of the stats for the stats_code

Re: How to build an alert based on status code?

DalJeanis — Wed, 09 Aug 2017 16:54:47 GMT

Huh? Oh, he wanted an alert when NONE of those events had been generated. Nope, still doesnt' work for me.

Ah, never mind, you posted the correct code on a new answer.

Here's the code for checking if ANY of those eleven event numbers had failed to be produced.

 index="YouShouldAlwaysSpecifyAnIndex" sourcetype="AndSourcetypeToo" status_code>=200 AND status_code<=210 
 | fields status_code
 | append [|stats count | eval status_code =mvrange(200,211) | mvexpand status_code |table status_code]
 | stats count BY status_code
 | eval count=count-1
 | where count=0

Re: How to build an alert based on status code?

woodcock — Wed, 09 Aug 2017 22:51:04 GMT

HA HA HA! I am an idiot! This is a "how many dogs didn't bark last night" problem!

Re: How to build an alert based on status code?

woodcock — Wed, 09 Aug 2017 23:00:17 GMT

Like this (for real this time):

index="YouShouldAlwaysSpecifyAnIndex" sourcetype="AndSourcetypeToo" status_code>=200 AND status_code<=210 
| appendpipe
    [| gentimes start=200 end=210 
    | streamstats count AS status_code
    | eval status_code = status_code + 199
    | fields status_code] 
| stats count BY status_code 
| eval count=count-1
| where count=0

Re: How to build an alert based on status code?

woodcock — Wed, 09 Aug 2017 23:01:25 GMT

The trick is that you cannot count events that didn't happen so you have to pretend that everything happened once to instantiate a counter for each one and then back down by 1.

Re: How to build an alert based on status code?

woodcock — Thu, 10 Aug 2017 13:46:17 GMT

If possible, Unaccept this answer and Accept my other one, which has the full/correct answer (which this one alludes to).

Re: How to build an alert based on status code?

ksharma7 — Wed, 03 Jun 2020 06:45:53 GMT

@woodcock can you explain this query please?

Re: How to build an alert based on status code?

ksharma7 — Wed, 03 Jun 2020 13:27:01 GMT

@woodcock can you please explain this query and what if I have to monitor only 200 status code not a range of codes..what modified query can work in that case?if you could help please

Re: How to build an alert based on status code?

ksharma7 — Wed, 03 Jun 2020 13:36:29 GMT

@vemurisurya can you explain the query you used to solve this problem and what if I have to monitor only one status code say 200 and want the same thing you stated