Splunk not applying time zone properly in clustered environment

[strive]

Hi,

As per Splunk documentation, Splunk applies time zone in the following order

1. Splunk Enterprise uses any time zone specified in raw event data (for example, PST, -0800).

2. Splunk Enterprise uses the value of a TZ attribute set in props.conf, if the event matches the host, source, or source type specified by the stanza.

3. If an event that arrives at an indexer originated at a forwarder, and both the forwarder and the receiving indexer run Splunk Enterprise 6.0 or later, then Splunk Enterprise uses the time zone that the forwarder provides.

In my clustered environment, it is always 3rd option. The first two has no effect in setting the timezone.

The props.conf file is present in all slave nodes under /appName/local/ directory

My props.conf settings are:

[mystanza]
TRUNCATE=0
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)\d{2}-[a-zA-Z]{3}-\d{4}\s\d{2}:\d{2}:\d{2}\s
MAX_TIMESTAMP_LOOKAHEAD=24
TIME_PREFIX=^
TIME_FORMAT=%d-%b-%Y %H:%M:%S %z

The log line is

20-Mar-2013 23:59:59 UTC DeviceName FileName|800|GET_PARAMETER rtsp://xx.xx.xx.xx:9100 RTSP/1.0

I even tried using TZ=UTC but no avail.

The forwarder is in IST and splunk always uses that timezone.

We have 2 log file sources and the time format is different in them. I have two stanzas in my props.conf under which we have defined TIME_FORMAT

Log file time: 20-Mar-2013 23:59:59 UTC

TIME_FORMAT is: %d-%b-%Y %H:%M:%S %z

Log file time: 2013-12-17 08:00:02.140310 UTC

TIME_FORMAT is: %Y-%m-%d %H:%M:%S.%6N %z

MAX_TIMESTAMP_LOOKAHEAD is set properly for these time formats.

Please let me know how to set the timezone as per log events.

Thanks

Strive

[yannK · Accepted solution]

There an error in your timeformat : day-month-year instead of year-month-day. Therefore the format is never matching

you want to try :
TIME_FORMAT=%Y-%b-%d %H:%M:%S %z

[strive]

When HWF is used, use props and tranforms on fwd. This works fine and solved our issues

[strive]

As per our tests, the following is the conclusion
1. While using heavy weight forwarder, If we set time on all our VMs as UTC then there is no issue in time zone setting for log events
2. If we use universal forwarder then we need not set time on our VMs as UTC. It can be local time zone. In this case whatever splunk documentation says about applying time zones, works properly.

[strive]

We have 2 log file sources and the time format is different in them. I have two stanzas in my props.conf under which we have defined TIME_FORMAT

Log file time: 20-Mar-2013 23:59:59 UTC
TIME_FORMAT is: %d-%b-%Y %H:%M:%S %z

Log file time: 2013-12-17 08:00:02.140310 UTC
TIME_FORMAT is: %Y-%m-%d %H:%M:%S.%6N %z

MAX_TIMESTAMP_LOOKAHEAD is set properly for these time formats.

[strive]

I am sorry. I put wrong log line there as example. I will edit my question.