topic Re: Alert triggering in Splunk due to slowness in Alerting

Alert triggering in Splunk due to slowness

logloganathan — Mon, 19 Mar 2018 10:57:10 GMT

we are alert in Splunk but when i checked, there is no issue.
as Splunk long time to search to query may be the reason.
Could anyone please give the suggestion

Re: Alert triggering in Splunk due to slowness

tiagofbmm — Mon, 19 Mar 2018 11:05:48 GMT

Can you show what search is the base for the alert?

Re: Alert triggering in Splunk due to slowness

logloganathan — Tue, 29 Sep 2020 18:32:01 GMT

actually its log event..
index=index_days sourcetype=sourcetype_name "search sring" | stats count

it will trigger alert if table value less than 1
but it triggering when there is no issue

Re: Alert triggering in Splunk due to slowness

logloganathan — Mon, 19 Mar 2018 11:55:30 GMT

waiting for the someone to provide the update

Re: Alert triggering in Splunk due to slowness

abhijit_mhatre — Mon, 19 Mar 2018 12:25:26 GMT

@logloganathan are you using any custom alert condition or are you using condition if number of results > 1?
Also, have you kept any throttling for the alert and do you want it to trigger it only once or for each result.

Let me know.

Re: Alert triggering in Splunk due to slowness

p_gurav — Mon, 19 Mar 2018 12:48:33 GMT

What do you mean by "it will trigger alert if table value less than 1"? Did you mean count < 1 in your search?

Re: Alert triggering in Splunk due to slowness

logloganathan — Mon, 19 Mar 2018 13:40:56 GMT

if number of results > 1 then only once

Re: Alert triggering in Splunk due to slowness

logloganathan — Mon, 19 Mar 2018 13:45:43 GMT

table query display lot of row
if it display no rows then i need alert

Re: Alert triggering in Splunk due to slowness

elliotproebstel — Mon, 19 Mar 2018 14:14:35 GMT

What's the time window the search is using? Depending on the delay for data populating through the system, a window that is too short/recent might alert even though data is in the index pipeline and shows up in later searches for the same time window.

Re: Alert triggering in Splunk due to slowness

logloganathan — Mon, 19 Mar 2018 15:01:40 GMT

i am using last 60 minutes

Re: Alert triggering in Splunk due to slowness

elliotproebstel — Mon, 19 Mar 2018 15:10:37 GMT

Ah, then that's not likely the issue. Is the search alerting every time it runs, or just sometimes? If it's every time, maybe the search is running with the wrong permissions or in the wrong app to actually gather the data expected.

Re: Alert triggering in Splunk due to slowness

logloganathan — Mon, 19 Mar 2018 15:41:14 GMT

search not altering everytime

Re: Alert triggering in Splunk due to slowness

logloganathan — Mon, 19 Mar 2018 16:05:33 GMT

waiting for the response.
Could anyone please update

Re: Alert triggering in Splunk due to slowness

elliotproebstel — Mon, 19 Mar 2018 16:28:48 GMT

If the search is failing to alert every single time, have you done the troubleshooting step of manually running the specific search as the user who is scheduled to run the alert inside the same app? I've made the mistake before of building an alert in one app and then saving/scheduling it in another and discovering it wasn't able to run as expected.

Re: Alert triggering in Splunk due to slowness

elliotproebstel — Tue, 20 Mar 2018 12:25:50 GMT

Can you tell us more about what you mean by "false alert is coming that is the issue due to slowness"? Are you saying your search takes so long to return that it times out, giving the false impression that there were no results? If so, maybe a solution would be to tune the query to ensure it never times out.

Based on your comments above, it seems like you're running this query:

index=index_days sourcetype=sourcetype_name "search sring" | stats count

And you want it to alert if there are 0 results returned, right? But you are getting alerts for times when you think it should have found results? If so, maybe try this:

index=index_days sourcetype=sourcetype_name "search sring" | head 1

That way, if there's a single result, it will find the first one and return immediately. That could help with a timeout.

Re: Alert triggering in Splunk due to slowness

logloganathan — Tue, 20 Mar 2018 12:51:19 GMT

wow exactly..same thing i want...Please enter the same in answer box

Re: Alert triggering in Splunk due to slowness

logloganathan — Tue, 20 Mar 2018 15:51:54 GMT

thanks for the answer

Re: Alert triggering in Splunk due to slowness

logloganathan — Tue, 29 Sep 2020 18:36:44 GMT

Could you please convert the same command to transforming command

index=index_days sourcetype=sourcetype_name "search sring" | head 1

Re: Alert triggering in Splunk due to slowness

elliotproebstel — Tue, 29 Sep 2020 18:35:36 GMT

Sure, but what's the goal of doing so? If we're just transforming for the sake of turning it into a table:

index=index_days sourcetype=sourcetype_name "search sring" 
| head 1
| stats values(*) AS *

or
index=index_days sourcetype=sourcetype_name "search sring"
| head 1
| stats count

Re: Alert triggering in Splunk due to slowness

DalJeanis — Wed, 21 Mar 2018 19:39:07 GMT

Hi @logloganathan. It seems like maybe you need some quicker feedback. For more direct help, please join the Splunk Slack channel via the form that is linked on the accepted answer on this page -
https://answers.splunk.com/answers/443734/is-there-a-splunk-slack-channel.html

On Slack, you can ask your question on the #n00b or #general channels, and people will chime in pretty quickly to help you.

Here, you can upvote any answers that you found particularly helpful. On the Slack channel, you can do something similar by typing @somebodysname++ (where somebodysname is their slack handle).