Alert triggering in Splunk due to slowness

[logloganathan]

we are alert in Splunk but when i checked, there is no issue.
as Splunk long time to search to query may be the reason.
Could anyone please give the suggestion

[DalJeanis]

Hi @logloganathan. It seems like maybe you need some quicker feedback. For more direct help, please join the Splunk Slack channel via the form that is linked on the accepted answer on this page -
https://answers.splunk.com/answers/443734/is-there-a-splunk-slack-channel.html

On Slack, you can ask your question on the #n00b or #general channels, and people will chime in pretty quickly to help you.

Here, you can upvote any answers that you found particularly helpful. On the Slack channel, you can do something similar by typing @somebodysname++ (where somebodysname is their slack handle).

[logloganathan]

Thanks DalJeanis

[elliotproebstel · Accepted solution]

Can you tell us more about what you mean by "false alert is coming that is the issue due to slowness"? Are you saying your search takes so long to return that it times out, giving the false impression that there were no results? If so, maybe a solution would be to tune the query to ensure it never times out.

Based on your comments above, it seems like you're running this query:

index=index_days sourcetype=sourcetype_name "search sring" | stats count

And you want it to alert if there are 0 results returned, right? But you are getting alerts for times when you think it should have found results? If so, maybe try this:

index=index_days sourcetype=sourcetype_name "search sring" | head 1

That way, if there's a single result, it will find the first one and return immediately. That could help with a timeout.

[logloganathan]

wow exactly..same thing i want...Please enter the same in answer box

[logloganathan]

thanks for the answer

[logloganathan]

Could you please convert the same command to transforming command

index=index_days sourcetype=sourcetype_name "search sring" | head 1

[elliotproebstel]

Sure, but what's the goal of doing so? If we're just transforming for the sake of turning it into a table:

index=index_days sourcetype=sourcetype_name "search sring" 
| head 1
| stats values(*) AS *

or
index=index_days sourcetype=sourcetype_name "search sring"
| head 1
| stats count

[tiagofbmm]

Can you show what search is the base for the alert?

[logloganathan]

actually its log event..
index=index_days sourcetype=sourcetype_name "search sring" | stats count

it will trigger alert if table value less than 1
but it triggering when there is no issue

[logloganathan]

waiting for the someone to provide the update

[abhijit_mhatre]

@logloganathan are you using any custom alert condition or are you using condition if number of results > 1?
Also, have you kept any throttling for the alert and do you want it to trigger it only once or for each result.

Let me know.

[logloganathan]

if number of results > 1 then only once

[elliotproebstel]

What's the time window the search is using? Depending on the delay for data populating through the system, a window that is too short/recent might alert even though data is in the index pipeline and shows up in later searches for the same time window.

[logloganathan]

i am using last 60 minutes

[elliotproebstel]

Ah, then that's not likely the issue. Is the search alerting every time it runs, or just sometimes? If it's every time, maybe the search is running with the wrong permissions or in the wrong app to actually gather the data expected.

[logloganathan]

search not altering everytime

[logloganathan]

waiting for the response.
Could anyone please update

[elliotproebstel]

If the search is failing to alert every single time, have you done the troubleshooting step of manually running the specific search as the user who is scheduled to run the alert inside the same app? I've made the mistake before of building an alert in one app and then saving/scheduling it in another and discovering it wasn't able to run as expected.

[p_gurav]

What do you mean by "it will trigger alert if table value less than 1"? Did you mean count < 1 in your search?

[logloganathan]

table query display lot of row
if it display no rows then i need alert