topic Re: Events have wrong timestamp. How to correct time configuration? in Getting Data In

Events have wrong timestamp. How to correct time configuration?

vetash — Fri, 15 Aug 2014 01:10:59 GMT

Hi all! Sorry, if this question was already asked by someone, but i'm stuck with a time configuration.
So, i just installed Splunk and configured it to listen on UDP port in my network. All hosts send data to it and everything is great, but Splunk shows the wrong time in search results.
This is how i see it:
https://www.dropbox.com/s/e4rf3kxete9qgpv/splunk_f.PNG
Also Splunk shows me the wrong time on all another hosts. Every time I type another ip - Splunk muss time.

This my date on server:
root@monsrv:~# date
Птн Авг 15 09:55:11 IRKT 2014

What do I need to configure to see the right time in search results?

Sorry for my bad English. Hope you understand me. 🙂

Re: Events have wrong timestamp. How to correct time configuration?

strive — Mon, 28 Sep 2020 17:20:51 GMT

Post your props.conf configurations.
What values you have set for TIME_FORMAT, TIME_PREFIX

Re: Events have wrong timestamp. How to correct time configuration?

vetash — Fri, 15 Aug 2014 04:59:36 GMT

Thanks for the reply!
/opt/splunk/etc/system/default/props.conf:
http://pastebin.com/pDzwZA6G

Re: Events have wrong timestamp. How to correct time configuration?

martin_mueller — Fri, 15 Aug 2014 07:21:57 GMT

The file in etc/system/default is useless to us because it only contains default values.

Re: Events have wrong timestamp. How to correct time configuration?

strive — Fri, 15 Aug 2014 08:05:07 GMT

Haven't you set your configurations in props.conf file. Your custom configurations should be under /etc/system/local. If you have written a separate app for heavy forwarder or indexer then the props.conf file should be under that app's local directory.

Re: Events have wrong timestamp. How to correct time configuration?

martin_mueller — Fri, 15 Aug 2014 09:14:18 GMT

Looking at the timestamps in your screenshot it seems this is a timezone issue. What time zone is the source and your user in? Your server seems to be in UTC+9?

Also, who's prepending the timestamp and host to the syslog event? Is your Splunk doing that, or is that already prepended before it gets to Splunk? If that's prepended before it gets to Splunk, what timezone is that system in?

Re: Events have wrong timestamp. How to correct time configuration?

vetash — Fri, 15 Aug 2014 23:45:29 GMT

Yes UTC+9. Timezone on system is right, also on the hosts who sending logs for splunk. On sceenshot on right side is actual date. Splunk shows incorrect date. (mark as circle 🙂 ) And i have no idea where i need to config it.

Re: Events have wrong timestamp. How to correct time configuration?

vetash — Fri, 15 Aug 2014 23:47:27 GMT

So where is required props.conf?
root@monsrv:~# find /opt/splunk/ -name props.conf

/opt/splunk/etc/apps/search/default/props.conf
/opt/splunk/etc/apps/legacy/default/props.conf
/opt/splunk/etc/apps/SplunkLightForwarder/default/props.conf
/opt/splunk/etc/apps/learned/local/props.conf
/opt/splunk/etc/apps/sample_app/default/props.conf
/opt/splunk/etc/system/default/props.conf

Re: Events have wrong timestamp. How to correct time configuration?

strive — Sat, 16 Aug 2014 07:19:25 GMT

This is direct from splunk documentation:

By default, Splunk Enterprise applies time zones using these rules, in this order:

Splunk Enterprise uses any time zone specified in raw event data (for example, PST, -0800).
Splunk Enterprise uses the value of a TZ attribute set in props.conf, if the event matches the host, source, or source type specified by the stanza.
If an event that arrives at an indexer originated at a forwarder, and both the forwarder and the receiving indexer run Splunk Enterprise 6.0 or later, then Splunk Enterprise uses the time zone that the forwarder provides.
Otherwise, Splunk Enterprise uses the time zone of the server that indexes the event.

Note: If you change the time zone setting in the system Splunk Enterprise runs on, you must restart Splunk Enterprise for it to pick up the change.

So in your case:

Point 1 is not applicable as your events do not contain time zone information.

Point 2 is also not applicable. Since you have not modified any props.conf settings. Also, you are not aware which props.conf contains the settings.

Then in that case either point 3 or point 4 is applicable. Since you have mentioned in your comment that both the host and the system(receiver) both are in UTC+9 timezone. That timezone is considered for indexing events.

What you need to do is this:

Step 1: Create props.conf file under /opt/splunk/etc/system/local/ directory of your indexer. The full path will look like this /opt/splunk/etc/system/local/props.conf on indexer node.
Note: You can also create this props.conf file under /opt/splunk/etc/apps/<your_app>/local/ directory. Here your_app is the dedicated app that you have created for your indexer node.

Step 2: Add a stanza with your sourcetype

[Your_Sourcetype]

Note: you can have stanza with source, host and sourcetype. I have chosen sourcetype here.

Step 3: Under that stanza specify the timezone

[Your_Sourcetype]
TZ = UTC

For more information on setting timezones read http://docs.splunk.com/Documentation/Splunk/6.1.3/Data/Applytimezoneoffsetstotimestamps

Re: Events have wrong timestamp. How to correct time configuration?

strive — Sat, 16 Aug 2014 07:22:24 GMT

I suggest you to read all these:
http://docs.splunk.com/Documentation/Splunk/6.1.3/Data/HowSplunkextractstimestamps
http://docs.splunk.com/Documentation/Splunk/6.1.3/Data/Configuretimestamprecognition
http://docs.splunk.com/Documentation/Splunk/6.1.3/Data/Configurepositionaltimestampextraction
http://docs.splunk.com/Documentation/Splunk/6.1.3/Data/Applytimezoneoffsetstotimestamps

Re: Events have wrong timestamp. How to correct time configuration?

martin_mueller — Tue, 19 Aug 2014 07:54:03 GMT

You should read this http://docs.splunk.com/Documentation/Splunk/6.1.3/Admin/Aboutconfigurationfiles and the following couple of pages.

Re: Events have wrong timestamp. How to correct time configuration?

vinodmadaan — Thu, 09 Apr 2015 08:46:41 GMT

HI Strive,

Thanks for your response, I am facing the same problem but with a weird twist that is this problem is not for all the records, I am having few records (about 100) that are having the this time stamp issue and rest are absolutely fine. Any idea what could possibly be the reason for that?

Thanks,
Vinod.