Getting Data In

Why are log sent to Splunk in GMT displayed in the future?

zliu
Splunk Employee
Splunk Employee

Several devices that only support sending logs out stamped with GMT and splunk displays them in the future.

Placed below in $SPLUNK_HOME/etc/system/local/props.conf

[host::snort*]
TZ = GMT

woodcock
Esteemed Legend

What is the "date_zone" value for your indexed events? If your setting is in effect, it should be "0000" or "UTC" or "GMT" but DEFINITELY NOT "local". My suspicion is that you will find it is "local", which tells you that your TZ setting is not in effect. Here are a few reasons this might be so:
1: If you are doing any kind of host-override (including "host_segment"), props.conf will trigger off the ORIGINAL host value, not the new host value; you might be able to use [source::] instead of [host::]
2: You might not be deploying the props.conf file to your indexers which is where this configuration file needs to be in order to function (NOTE: there is a twist to this as of v6.0 in that a props.conf TZ setting on the forwarder will be honored and have precedence over the same setting on the indexer)

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

If the Splunk indexer's clock is correct, and the Splunk indexer's time zone is correct, then in fact all you should need to do it tell Splunk what the time zone of your incoming data is. Splunk will do the appropriate corrections.

If your stanza:

[host::snort*]
TZ=GMT

is not working, it is likely because either you have configured it in the wrong place (it must be where the parsing is done: http://www.splunk.com/wiki/Where_do_I_configure_my_Splunk_settings%3F), or because the host name does not (yet) match. e.g., for syslog inputs, the host value may not match what you see in Splunk because Splunk has transformed it.

gkanapathy
Splunk Employee
Splunk Employee

Are you saying that the [host::snort*] stanza you used is not working to adjust the time appropriately? Please clarify.

0 Karma

Mick
Splunk Employee
Splunk Employee

Splunk extracts timestamps from events and indexes them in epoch format, and uses UTC time (Coordinated Universal Time). To see what time Splunk has applied to a particular event, check out the '_time' field.

When you apply an offset, Splunk will determine the timezone that your server is running in and increase/decrease the '_time' value appropriately. When searching this data, Splunk will display the time relative to your current location. Eg, an event from 16:00 GMT will be displayed as 08:00 PST

If your data is appearing with a time in the future next to it, it's possible that the indexing server is set to the wrong timezone, or perhaps Splunk is extracting a different timestamp than you expect it to.

If you can supply an example of the event time vs. the indexed time, and maybe a sample event, perhaps we can narrow this down further. Always provide as much information as possible with your question, the less we go back & forth the better.

Chris_R_
Splunk Employee
Splunk Employee

lulz at typing simultaneous response

0 Karma

Chris_R_
Splunk Employee
Splunk Employee

Hey Splunker_J, Which timezone is the forwarding device and indexer in?

it's possible the snort timestamps are just not being parsed correctly. Do you have any sample event data from them? You might have to specify the time string with:

TIME_PREFIX = < some characters >
TIME_FORMAT = %d/%b/%Y:%H:%M:%S this would match a date like 03/Feb/2010:12:35:11

(For TIME_FORMAT syntax use a reference for strptime such as http://www.opengroup.org/onlinepubs/009695399/functions/strptime.html

Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

Industry Solutions for Supply Chain and OT, Amazon Use Cases, Plus More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...