Getting Data In

Splunk Time stamp Error

clyde772
Communicator

Hey Gurus!

I am processing F/W log such as below which I recieve through syslog server.

2010-06-29T20:48:26.742950+10:00 10.91.30.1 10110decf42010062920: 49:10Deny17090223182612192.168.100.1102410.91.50.43514eth0eth5````1971```

I am using rsyslogd daemon to process syslog packets, and somehow in the date field it looks like this,

2010-06-29T20:48:26

You realize that there is "T" in between the date and time. I don;t know if this is the cause but Splunk seems to process the date as 1 hour behind.

2010-06-29 07:48:26 PM in splunk.

Instead of

2010-06-29 08:48:26 PM

Have anybody seen similar problem? How can I fix it?

YhC.

Tags (1)

blebit
Path Finder

hello, can we push this from Deployment Monitor ??

0 Karma

mitch
Explorer

I agree with Lowell above -- if it's exactly an hour off it's probably not a time extraction issue, but a timezone issue.
Info that would be useful:
* Do a search that finds that single event and then add "| eval t=_time | table t" to the search. You should see the raw UNIX epoch time (i.e. seconds since Midnight 1/1/1970, GMT). What number do we report that for that event. It'll be a number around 1.2 billion.
* What OS are you running?
* What version of Splunk?
* What timezone are you in? How specifically is it configured in the OS?

0 Karma

Lowell
Super Champion

If you haven't tried using an explicit TIME_FORMAT in your props.conf file for this sourcetype, then I would suggest that you try the following:

TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%z

If this doesn't work for you, then you may need to hardcode a timezone using the TZ setting in your props.conf file. If you end up using a custom timezone, then you may need to remove the %z from the above, I'm not sure how that works exactly. Another consideration is this: With the daylight savings changes (aka timezone changes) in the last couple of years, it's possible that whatever app that is creating this log has an old timezone table that could be out of date--just another possibility to consider.

Docs:

hiwell
Explorer

To answer Lowell's question I believe Splunk is extracting the time automatically. If he is specifying the TIME_FORMAT the adding of 1hour can be avoided easily I believe - by just writing the regular expression

You could extract the time manually by adding a few things in the props.conf file. Something like

[<sourcetype>]
TIME_FORMAT=%Y-%m-%dT%H:%M:%S

I just did the same thing yesterday because Splunk wasn't extracting the timestamps correctly

more info here http://www.splunk.com/base/Documentation/3.0.2/Admin/Propsconf
and here about the time formatting http://apidock.com/ruby/DateTime/strftime

0 Karma

hiwell
Explorer

Hah you probably know whats going on but you're getting me confused here.. I meant to say that you have to keep the T in timestamp if you use the TIME_FORMAT I provided in the above comment. Is this what you're trying to say.

0 Karma

Lowell
Super Champion

No, actually you can remove the "T" with a regex, but that kind of transformation happens after timestamp recognition so you would have keep the "T" in TIME_FORMAT.... But my point was that you can't modify the timestamp with a regex prior to the timestamping process and even if you could, you can't use a regex to do arithmetic; thus you can't change the time +/- an hour.

0 Karma

hiwell
Explorer

Yeah you can't remove that T with it.

0 Karma

Lowell
Super Champion

don't think its quite that simple. There is really nothing you can do with a regex to solve this problem; you can't add/subtract with a regex. Also, since the log file includes some timezone info, we should probably attempt to use it in TIME_FORMAT if at all possible, unless it's buggy. (BTW, the splunk docs you linked to (3.0.2) are very old at this point, 4.1.3 is the "latest" right now.)

0 Karma

Lowell
Super Champion

What's your props.conf entry for this source/sourcetype? Do you have a TIME_FORMAT specified, or is splunk automatically extracting timestamps?

0 Karma

Lowell
Super Champion

Does your event literally contain a bunch of backticks ("`"), or is that for formatting your event? (Try indenting your sample by 4 spaces if you want to capture a literal string.

0 Karma

Lowell
Super Champion

Are you sure this isn't a timezone issue? Are you +10? if you could post the value of date_zone field for your indexed events, that would help.

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!