Alerting

Splunk alert no results

sonila
Path Finder

alt text

I have made an email alert. but when i click to view events on triggered alert i see no results. why this happens? how can i show the results?

Tags (3)
0 Karma

woodcock
Esteemed Legend

The only thing that I can think of is that your events are expiring between when the alert hits and when you double-check. This should tell you what the oldest event still in the index is. It should be weeks, if not months old but maybe it is hours or days old.

|metadata type=sourcetypes | search sourcetype=log4net
0 Karma

woodcock
Esteemed Legend

Usually when I have stuff that "tests OK" in an ad-hoc search but fails in a scheduled search it is due to pipeline latency. Check out the values of _indextime - _time for your events. These should be positive and no more than 300ish.

0 Karma

sonila
Path Finder

And what do you recommend after checking the values of _indextime - _time
_indextime - _time is less than 0 to my indexed data. What should i do?
_indextime - _time is around -9.581

0 Karma

woodcock
Esteemed Legend

Since the magnatude is so low, the problem is surely that your forwarders and/or indexers are not using NTP and have drifted from true. To see if it is your indexers, try this:

| rest /services/server/info 
| eval updated_t=round(strptime(updated, "%Y-%m-%dT%H:%M:%S%z")) 
| eval delta_t=now()-updated_t 
| eval delta=tostring(abs(delta_t), "duration") 
| table serverName, updated, updated_t, delta, delta_t

If delta is anything other than about 00:00:01 (which is easy to account for when processing a lot of indexers), you have clock skew and are a naughty boy because you should have setup NTP on your indexers.

NOTE: this IS a problem, but it is not the problem that you were asking about.

0 Karma

sonila
Path Finder

delta is 00:00:00 and _indextime - _time is around 9.581 it is positive

0 Karma

woodcock
Esteemed Legend

In that case never mind this whole answer.

0 Karma

sonila
Path Finder

can you help me about my problem why i dont see results in splunk?

0 Karma

kmaron
Motivator

Your trigger times in the capture show 12:27 to 12:34 but your search shows 1:11 to 1:21. Is it possible that there were no triggered events between 1:11 and 1:21? What if you change your search time frame to the 12:27-12:34?

0 Karma

sonila
Path Finder

I tried but no result again

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Announcing Modern Navigation: A New Era of Splunk User Experience

We are excited to introduce the Modern Navigation feature in the Splunk Platform, available to both cloud and ...

Modernize your Splunk Apps – Introducing Python 3.13 in Splunk

We are excited to announce that the upcoming releases of Splunk Enterprise 10.2.x and Splunk Cloud Platform ...

Step into “Hunt the Insider: An Splunk ES Premier Mystery” to catch a cybercriminal ...

After a whole week of being on call, you fell asleep on your keyboard, and you hit a sequence of buttons that ...