topic Re: Splunk alert no results in Alerting

Splunk alert no results

sonila — Tue, 09 May 2017 11:43:59 GMT

I have made an email alert. but when i click to view events on triggered alert i see no results. why this happens? how can i show the results?

Re: Splunk alert no results

kmaron — Tue, 09 May 2017 13:20:13 GMT

Your trigger times in the capture show 12:27 to 12:34 but your search shows 1:11 to 1:21. Is it possible that there were no triggered events between 1:11 and 1:21? What if you change your search time frame to the 12:27-12:34?

Re: Splunk alert no results

sonila — Tue, 09 May 2017 21:55:46 GMT

I tried but no result again

Re: Splunk alert no results

woodcock — Wed, 10 May 2017 16:20:49 GMT

Usually when I have stuff that "tests OK" in an ad-hoc search but fails in a scheduled search it is due to pipeline latency. Check out the values of _indextime - _time for your events. These should be positive and no more than 300ish.

Re: Splunk alert no results

sonila — Wed, 10 May 2017 20:15:53 GMT

And what do you recommend after checking the values of _indextime - _time
_indextime - _time is less than 0 to my indexed data. What should i do?
_indextime - _time is around -9.581

Re: Splunk alert no results

woodcock — Thu, 11 May 2017 20:39:38 GMT

Since the magnatude is so low, the problem is surely that your forwarders and/or indexers are not using NTP and have drifted from true. To see if it is your indexers, try this:

| rest /services/server/info 
| eval updated_t=round(strptime(updated, "%Y-%m-%dT%H:%M:%S%z")) 
| eval delta_t=now()-updated_t 
| eval delta=tostring(abs(delta_t), "duration") 
| table serverName, updated, updated_t, delta, delta_t

If delta is anything other than about 00:00:01 (which is easy to account for when processing a lot of indexers), you have clock skew and are a naughty boy because you should have setup NTP on your indexers.

NOTE: this IS a problem, but it is not the problem that you were asking about.

Re: Splunk alert no results

sonila — Thu, 11 May 2017 21:39:24 GMT

delta is 00:00:00 and _indextime - _time is around 9.581 it is positive

Re: Splunk alert no results

woodcock — Thu, 11 May 2017 22:23:55 GMT

In that case never mind this whole answer.

Re: Splunk alert no results

sonila — Fri, 12 May 2017 07:27:06 GMT

can you help me about my problem why i dont see results in splunk?

Re: Splunk alert no results

woodcock — Fri, 12 May 2017 22:40:20 GMT

The only thing that I can think of is that your events are expiring between when the alert hits and when you double-check. This should tell you what the oldest event still in the index is. It should be weeks, if not months old but maybe it is hours or days old.

|metadata type=sourcetypes | search sourcetype=log4net