Getting Data In
Highlighted

Events have wrong timestamp. How to correct time configuration?

New Member

Hi all! Sorry, if this question was already asked by someone, but i'm stuck with a time configuration.
So, i just installed Splunk and configured it to listen on UDP port in my network. All hosts send data to it and everything is great, but Splunk shows the wrong time in search results.
This is how i see it:
https://www.dropbox.com/s/e4rf3kxete9qgpv/splunk_f.PNG
Also Splunk shows me the wrong time on all another hosts. Every time I type another ip - Splunk muss time.

This my date on server:
root@monsrv:~# date
Птн Авг 15 09:55:11 IRKT 2014

What do I need to configure to see the right time in search results?

Sorry for my bad English. Hope you understand me. 🙂

0 Karma
Highlighted

Re: Events have wrong timestamp. How to correct time configuration?

Influencer

Post your props.conf configurations.
What values you have set for TIMEFORMAT, TIMEPREFIX

0 Karma
Highlighted

Re: Events have wrong timestamp. How to correct time configuration?

New Member

Thanks for the reply!
/opt/splunk/etc/system/default/props.conf:
http://pastebin.com/pDzwZA6G

0 Karma
Highlighted

Re: Events have wrong timestamp. How to correct time configuration?

SplunkTrust
SplunkTrust

The file in etc/system/default is useless to us because it only contains default values.

Highlighted

Re: Events have wrong timestamp. How to correct time configuration?

Influencer

Haven't you set your configurations in props.conf file. Your custom configurations should be under /etc/system/local. If you have written a separate app for heavy forwarder or indexer then the props.conf file should be under that app's local directory.

0 Karma
Highlighted

Re: Events have wrong timestamp. How to correct time configuration?

SplunkTrust
SplunkTrust

Looking at the timestamps in your screenshot it seems this is a timezone issue. What time zone is the source and your user in? Your server seems to be in UTC+9?

Also, who's prepending the timestamp and host to the syslog event? Is your Splunk doing that, or is that already prepended before it gets to Splunk? If that's prepended before it gets to Splunk, what timezone is that system in?

0 Karma
Highlighted

Re: Events have wrong timestamp. How to correct time configuration?

New Member

Yes UTC+9. Timezone on system is right, also on the hosts who sending logs for splunk. On sceenshot on right side is actual date. Splunk shows incorrect date. (mark as circle 🙂 ) And i have no idea where i need to config it.

0 Karma
Highlighted

Re: Events have wrong timestamp. How to correct time configuration?

New Member

So where is required props.conf?
root@monsrv:~# find /opt/splunk/ -name props.conf

/opt/splunk/etc/apps/search/default/props.conf
/opt/splunk/etc/apps/legacy/default/props.conf
/opt/splunk/etc/apps/SplunkLightForwarder/default/props.conf
/opt/splunk/etc/apps/learned/local/props.conf
/opt/splunk/etc/apps/sample_app/default/props.conf
/opt/splunk/etc/system/default/props.conf

0 Karma
Highlighted

Re: Events have wrong timestamp. How to correct time configuration?

Influencer

This is direct from splunk documentation:

By default, Splunk Enterprise applies time zones using these rules, in this order:

  1. Splunk Enterprise uses any time zone specified in raw event data (for example, PST, -0800).

  2. Splunk Enterprise uses the value of a TZ attribute set in props.conf, if the event matches the host, source, or source type specified by the stanza.

  3. If an event that arrives at an indexer originated at a forwarder, and both the forwarder and the receiving indexer run Splunk Enterprise 6.0 or later, then Splunk Enterprise uses the time zone that the forwarder provides.

  4. Otherwise, Splunk Enterprise uses the time zone of the server that indexes the event.

Note: If you change the time zone setting in the system Splunk Enterprise runs on, you must restart Splunk Enterprise for it to pick up the change.

So in your case:

Point 1 is not applicable as your events do not contain time zone information.

Point 2 is also not applicable. Since you have not modified any props.conf settings. Also, you are not aware which props.conf contains the settings.

Then in that case either point 3 or point 4 is applicable. Since you have mentioned in your comment that both the host and the system(receiver) both are in UTC+9 timezone. That timezone is considered for indexing events.

What you need to do is this:

Step 1: Create props.conf file under /opt/splunk/etc/system/local/ directory of your indexer. The full path will look like this /opt/splunk/etc/system/local/props.conf on indexer node.
Note: You can also create this props.conf file under /opt/splunk/etc/apps/<your_app>/local/ directory. Here your_app is the dedicated app that you have created for your indexer node.

Step 2: Add a stanza with your sourcetype

[Your_Sourcetype]

Note: you can have stanza with source, host and sourcetype. I have chosen sourcetype here.

Step 3: Under that stanza specify the timezone

[Your_Sourcetype]
TZ = UTC

For more information on setting timezones read http://docs.splunk.com/Documentation/Splunk/6.1.3/Data/Applytimezoneoffsetstotimestamps

0 Karma