Splunk Time stamp Error

[clyde772]

Hey Gurus!

I am processing F/W log such as below which I recieve through syslog server.

2010-06-29T20:48:26.742950+10:00 10.91.30.1 10110decf42010062920: 49:10Deny17090223182612192.168.100.1102410.91.50.43514eth0eth5````1971```

I am using rsyslogd daemon to process syslog packets, and somehow in the date field it looks like this,

2010-06-29T20:48:26

You realize that there is "T" in between the date and time. I don;t know if this is the cause but Splunk seems to process the date as 1 hour behind.

2010-06-29 07:48:26 PM in splunk.

Instead of

2010-06-29 08:48:26 PM

Have anybody seen similar problem? How can I fix it?

YhC.

[blebit]

hello, can we push this from Deployment Monitor ??

[mitch]

I agree with Lowell above -- if it's exactly an hour off it's probably not a time extraction issue, but a timezone issue.
Info that would be useful:
* Do a search that finds that single event and then add "| eval t=_time | table t" to the search. You should see the raw UNIX epoch time (i.e. seconds since Midnight 1/1/1970, GMT). What number do we report that for that event. It'll be a number around 1.2 billion.
* What OS are you running?
* What version of Splunk?
* What timezone are you in? How specifically is it configured in the OS?

[Lowell]

If you haven't tried using an explicit TIME_FORMAT in your props.conf file for this sourcetype, then I would suggest that you try the following:

TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%z

If this doesn't work for you, then you may need to hardcode a timezone using the TZ setting in your props.conf file. If you end up using a custom timezone, then you may need to remove the %z from the above, I'm not sure how that works exactly. Another consideration is this: With the daylight savings changes (aka timezone changes) in the last couple of years, it's possible that whatever app that is creating this log has an old timezone table that could be out of date--just another possibility to consider.

Docs:

• Specify timezones of timestamps

[hiwell]

To answer Lowell's question I believe Splunk is extracting the time automatically. If he is specifying the TIME_FORMAT the adding of 1hour can be avoided easily I believe - by just writing the regular expression

You could extract the time manually by adding a few things in the props.conf file. Something like

[<sourcetype>]
TIME_FORMAT=%Y-%m-%dT%H:%M:%S

I just did the same thing yesterday because Splunk wasn't extracting the timestamps correctly

more info here http://www.splunk.com/base/Documentation/3.0.2/Admin/Propsconf
and here about the time formatting http://apidock.com/ruby/DateTime/strftime

[hiwell]

Hah you probably know whats going on but you're getting me confused here.. I meant to say that you have to keep the T in timestamp if you use the TIME_FORMAT I provided in the above comment. Is this what you're trying to say.

[Lowell]

No, actually you can remove the "T" with a regex, but that kind of transformation happens after timestamp recognition so you would have keep the "T" in TIME_FORMAT.... But my point was that you can't modify the timestamp with a regex prior to the timestamping process and even if you could, you can't use a regex to do arithmetic; thus you can't change the time +/- an hour.

[hiwell]

Yeah you can't remove that T with it.

[Lowell]

don't think its quite that simple. There is really nothing you can do with a regex to solve this problem; you can't add/subtract with a regex. Also, since the log file includes some timezone info, we should probably attempt to use it in TIME_FORMAT if at all possible, unless it's buggy. (BTW, the splunk docs you linked to (3.0.2) are very old at this point, 4.1.3 is the "latest" right now.)

[Lowell]

What's your props.conf entry for this source/sourcetype? Do you have a TIME_FORMAT specified, or is splunk automatically extracting timestamps?

[Lowell]

Does your event literally contain a bunch of backticks ("`"), or is that for formatting your event? (Try indenting your sample by 4 spaces if you want to capture a literal string.

[Lowell]

Are you sure this isn't a timezone issue? Are you +10? if you could post the value of date_zone field for your indexed events, that would help.