topic Re: postfix_syslog time extraction inaccuracies in Splunk Search

postfix_syslog time extraction inaccuracies

adamw — Wed, 29 Sep 2010 19:22:44 GMT

We seem to be having an issue with the postfix_syslog sourcetype (that came as a default sourcetype in Splunk) and its date extractions.

I posted this at 8:20am on 9/29, and did a search of events that take place between 15:00 and 23:59 on 9/29 and come back with the following results.

As you can see, the date_hour is set up as 18 on one of these events, which translates to 6pm, but the original event actually took place at 5am.

I am not overriding any of the default postfix_syslog stuff, and these events are getting sourcetyped properly, as shown below.

[postfix_syslog]
TIME_FORMAT = %b %d %H:%M:%S

According to the docs, %H is the 24 hour time, even though Splunk seems to believe it is not.

Any help is appreciated. Thanks,

--adam (EDIT) This is Splunk 4.1.4, data is coming in with syslog-ng.

Re: postfix_syslog time extraction inaccuracies

Lowell — Thu, 30 Sep 2010 05:35:29 GMT

That's very weird. Some additional details may help. Please "edit" your post and provide: The version of splunk you are running. Have you ever made any changes to datetime.xml? Does the syslog host contain any digits it or is it an IP address?

Re: postfix_syslog time extraction inaccuracies

southeringtonp — Thu, 30 Sep 2010 06:06:59 GMT

Check to make sure that there isn't another rule overriding TIME_FORMAT or other timestamping options. If you have conflicting entries in props.conf, configuration settings applied to [host::myhost] or '[source::mysource] will take precedence over those applied to the sourcetype.

If that isn't the problem, then more information is always helpful, such as:

How are you bringing the syslog data in? (Looks like syslog-ng?)

How is the input configured in inputs.conf?

Re: postfix_syslog time extraction inaccuracies

gkanapathy — Thu, 30 Sep 2010 11:25:12 GMT

Seems likely that you have some conflicting configuration. Splunk does not appear to be looking at your event timestamps at all, but using CURRENT or the file modification time. This may be because of changes or redefinitions of DATETIME_CONFIG or the file it points to. Probably using btool http://www.splunk.com/base/Documentation/4.1.5/Admin/Troubleshootingconfigurations may help show if this is so, of if there are other configurations conflicting (e.g., both source:: and host:: configurations will override sourcetype props.conf configurations.)

Re: postfix_syslog time extraction inaccuracies

Lowell — Thu, 30 Sep 2010 20:55:40 GMT

One more possibly helpful resource:

What’s the best way to track down props.conf problems?

Re: postfix_syslog time extraction inaccuracies

Lowell — Thu, 30 Sep 2010 21:09:16 GMT

I agree, a rogue DATETIME_CONFIG = CURRENT entry does seem to be the most logical explication for what's going on here.