topic Re: How to set timezone in an advanced configuration? in Getting Data In

How to set timezone in an advanced configuration?

hulahoop — Sat, 20 Mar 2010 08:51:46 GMT

If I have a basic input which sets the sourcetype, configuring a timezone offset works great:

In inputs.conf:

[monitor:///path/to/foo.log]
sourcetype = foo

In props.conf:

[foo]
TZ = GMT

If I have to setup flexible sourcetyping, the above configuration does not work:

In inputs.conf:

[monitor:///path/to/foo.log]

In props.conf:

[source::...foo.log]
TRANSFORMS-abc = setFooSourcetype, setBarSourcetype

[foo]
TZ = GMT

I'm guessing the timezone is set before the new sourcetype is applied so that is why the TZ parameter is not honored. So I then tried to set the timezone offset using host, which also does not work (no matter the ordering of the stanzas):

[source::...foo.log]
TRANSFORMS-abc = setFooSourcetype

[host::foohost*]
TZ = GMT

What other options do I have? I'm not sure where in the indexing pipeline metadata is set.

Re: How to set timezone in an advanced configuration?

gkanapathy — Sun, 21 Mar 2010 01:56:44 GMT

The TZ parameter is used and set by the parsing pipeline: http://www.splunk.com/wiki/Where_do_I_configure_my_Splunk_settings%3F and I believe that timestamp processing occurs before any TRANSFORMS. Setting TZ with a host:: or source:: stanza should work (as it does with a sourcetype), so I'm not sure why it isn't working for you. Is there another stanza that might also be matching and overriding?

Re: How to set timezone in an advanced configuration?

hulahoop — Mon, 22 Mar 2010 09:30:49 GMT

I think that is precisely the problem--"timestamp processing occurs before any TRANSFORMS." This data stream requires a sourcetype and host override. Is there a way to re-process TZ after the sourcetype and host override? Can I create my own processor for this or edit the pipeline order for the parsing queue?

More details: all events are streamed via TCP by a syslog server, except the events are not in syslog format. So we need to add index-time rules to assign sourcetype and host.

Re: How to set timezone in an advanced configuration?

gkanapathy — Mon, 22 Mar 2010 21:24:09 GMT

oh, okay, yes if you're setting host in props/transforms, then no stanza referring to them is going to work (by that host). Are all events from multiple hosts coming in from that same syslog server? Is it possible to split the hosts from different time zones to different syslog servers or different TCP ports?

Re: How to set timezone in an advanced configuration?

hulahoop — Tue, 23 Mar 2010 00:24:23 GMT

Thank you, G. We are still in the eval stage here so are reluctant to make changes like this in the production environment. The preference is to configure Splunk to handle this case if possible. Are you recommending we do not alter the parsing pipeline?

Re: How to set timezone in an advanced configuration?

jrodman — Thu, 25 Mar 2010 02:16:20 GMT

You really want to declare the TZ by host, ideally, since the logs are almost certainly generated by some system in either its localtime or in GMT.

However, sometimes life is more complicated than that, like syslog, where the host is identified via a transform, and the original host (used during timestamp extraction) is going to just be where we're acquiring the data.

In this case you're going to just have to use a source pattern to get reasonable behavior, with the hosts split out into files, for example by syslog-ng.

Of course if you have the option of simply altering the timestamp format to include the timezone, that's really ideal for ALL parties, not just Splunk.

Re: How to set timezone in an advanced configuration?

hulahoop — Wed, 14 Apr 2010 05:31:37 GMT

In the current release of Splunk, in the exact scenario described above, it is not possible to apply TZ when using host/sourcetype overriding. The only alternative is to have the specific hosts forward directly to Splunk so as to create a dedicated source.

Re: How to set timezone in an advanced configuration?

vbumgarner — Wed, 22 Dec 2010 04:09:10 GMT

I just ran into this problem. 😞

Re: How to set timezone in an advanced configuration?

gfriedmann — Sat, 26 Feb 2011 00:51:56 GMT

Man, it would be nice if Splunk's WinEventLog inputs included timezone information.

Re: How to set timezone in an advanced configuration?

jrodman — Wed, 02 Mar 2011 02:23:29 GMT

If the timezone is declared at time of parsing, it is stored. However, the date parsing code essentially passes the buck for localtime to the system libc, so doesn't know it. Getting the offset is some work. Getting the timezone is pretty hard. However, this sounds like work worth doing. It would be very helpful if you could file an enhancement request with support with one or two use cases to add color to the need.

Re: How to set timezone in an advanced configuration?

lguinn2 — Sat, 03 Sep 2011 18:07:47 GMT

Cross-referencing @Sorkin's answer to a timezone question.