Alerting

Set up splunk Alert - a complicated one

loveforsplunk
Explorer

Here is my log file having a key word "error":

My search is : index=abc host="123" "error" source="efg/*"

My search returns results as below (check out the timings ) , suppose below is the event section:

Time Event
5/20/17 1:00:45.000 AM Completed at Sat May 20 03:00:45 2017

Under the _time section , the time which is displayed is 2hrs less than the time that is displayed in the logs(as you can see from the event section).

Now , suppose there is a failure in my log which I came to know right now. I go to splunk and check I do not see any result for the last 15 minutes even but when I do last 2 hours , I get to see the result.

Please tell me how do i set this alert. if I am setting to check every 5 minutes, I do not get any alert . When I did -2h as start time and now as finish time , I still did not get any alert . Now I did it -2h@h which I am sure will work or not until there is a failure.

Also, I have selected Run every minute while setting the alert.

Tags (1)
0 Karma
1 Solution

dineshraj9
Builder

You need to fix the timezone of your logs first. The 2hr difference is because of the logs being set to incorrect timezone.

https://docs.splunk.com/Documentation/Splunk/6.6.0/Data/Applytimezoneoffsetstotimestamps

I suggest you get the TZ attribute in props.conf for the sourcetype.
This will fix the timezone issue and logs will appear real time.

View solution in original post

0 Karma

dineshraj9
Builder

You need to fix the timezone of your logs first. The 2hr difference is because of the logs being set to incorrect timezone.

https://docs.splunk.com/Documentation/Splunk/6.6.0/Data/Applytimezoneoffsetstotimestamps

I suggest you get the TZ attribute in props.conf for the sourcetype.
This will fix the timezone issue and logs will appear real time.

0 Karma

loveforsplunk
Explorer

But for the same server , I have alert set up for other logs as well. For those I get the expected output . Only for this particular log I do not get, not sure why.

Do u mean , there is a way to set up timezone for particular logs ??

In my Splunk user settings , my timezone set is correct and the logs timezone in its server as I see is also the same as mine , then why do I get something else in _time ?

0 Karma

dineshraj9
Builder

Yes, you can setup for timezone for each log in different ways.

In props.conf, you can set TZ attribute for the particular sourcetype.

In inputs.conf, you can set _tzhint field for the particular log monitor stanza.

0 Karma

loveforsplunk
Explorer

oh ok. I will work on this. Thank you so much Dinesh.

0 Karma

adonio
Ultra Champion

it has nothing to do with your users timezone,
check the link in the answer

0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Get the T-shirt to Prove You Survived Splunk University Bootcamp

As if Splunk University, in Las Vegas, in-person, with three days of bootcamps and labs weren’t enough, now ...

Wondering How to Build Resiliency in the Cloud?

IT leaders are choosing Splunk Cloud as an ideal cloud transformation platform to drive business resilience,  ...