Getting Data In

Why are the timestamps for events indexed in Splunk behind by 5 hours compared to the original logs?

Contributor

Splunk is not showing the correct time that logs are coming in. They are behind by five hours. The time on the server is correct and so are the logs themselves, but Splunk's time is not correct.

My log that splunk is saying came in at 10:29AM-

1 2015-02-19T15:20:29.916Z Alert - "SOME TEXT FROM LOG" timestamp="2015-02-19 15:17:43 EST" LOGSTUFF="Logout completed by user from 8.8.8.8"

I have made changes to the props.conf files on our indexers with no luck. Any one run into this issue before?

1 Solution

Contributor

Splunk support helped me out on this, i had to add the stanza on the heavy forwarder and not the indexers

[host::<yourHost>]
TIME_PREFIX = ^1\s  
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
MAX_TIMESTAMP_LOOKAHEAD = 30

thanks everyone for helping me out!

View solution in original post

0 Karma

Contributor

Splunk support helped me out on this, i had to add the stanza on the heavy forwarder and not the indexers

[host::<yourHost>]
TIME_PREFIX = ^1\s  
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
MAX_TIMESTAMP_LOOKAHEAD = 30

thanks everyone for helping me out!

View solution in original post

0 Karma

SplunkTrust
SplunkTrust

That is somewhat of a given. Anything that is acquired as part of event parsing (timestamp recognition, host name, etc) must ALWAYS be done on the first parsing node to touch the data. Indexers are parsing nodes, as are heavy forwarders.

As a best practice, if you have something in your topology outside of indexers and universal forwarders, I would suggest making that information part of your question to begin with. Also, ESR's http://www.catb.org/esr/faqs/smart-questions.html is always a good read to help remind you how to focus technical questions in a way that helps make the community able to best assist.

SplunkTrust
SplunkTrust

It's probably the "Z" at the end of your time. 2015-02-19T15:20:29.916Z is equivalent (this time of year) to 10:20:29 EST. Even though you have "told" Splunk that the server is in EST, having a time in your event with a TZ indicator on the end like a "Z" will get you a timezone offset. You can probably get closer to confirming this by using time timestartpos and timeendpos fields that Splunk creates and see what character positions it occupies.

If your application is writing out a timestamp in local time and then the wrong putting timezone indicators on it, then your app needs to be corrected.

Alternately, if you can't fix the application to not write logs in this semi-derpy way the following props.conf entries might help:

TIME_PREFIX = ^1\s
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N
MAX_TIMESTAMP_LOOKAHEAD = 23

Contributor

I have tried a few different things in props.conf right now i have it set as-

MAX_TIMESTAMP_LOOKAHEAD=150
NO_BINARY_CHECK=1
TIME_PREFIX=\1\s

I have also tried changing the time zone with no luck. Also, it should be EST.

0 Karma

SplunkTrust
SplunkTrust

It would help to see the relevant props.conf stanza. Unless there's a typo in your message, it looks like Splunk and the log agree that the entry occurred at 10:20-ish AM. There's some room for confusion, however, since the log message has the same timestamp (within 3 minutes) in two different time zones.

---
If this reply helps you, an upvote would be appreciated.
0 Karma