Getting Data In

How to set timezone in an advanced configuration?

hulahoop
Splunk Employee
Splunk Employee

If I have a basic input which sets the sourcetype, configuring a timezone offset works great:

In inputs.conf:

[monitor:///path/to/foo.log]
sourcetype = foo

In props.conf:

[foo]
TZ = GMT

If I have to setup flexible sourcetyping, the above configuration does not work:

In inputs.conf:

[monitor:///path/to/foo.log]

In props.conf:

[source::...foo.log]
TRANSFORMS-abc = setFooSourcetype, setBarSourcetype

[foo]
TZ = GMT

I'm guessing the timezone is set before the new sourcetype is applied so that is why the TZ parameter is not honored. So I then tried to set the timezone offset using host, which also does not work (no matter the ordering of the stanzas):

[source::...foo.log]
TRANSFORMS-abc = setFooSourcetype

[host::foohost*]
TZ = GMT

What other options do I have? I'm not sure where in the indexing pipeline metadata is set.

Tags (2)
1 Solution

hulahoop
Splunk Employee
Splunk Employee

In the current release of Splunk, in the exact scenario described above, it is not possible to apply TZ when using host/sourcetype overriding. The only alternative is to have the specific hosts forward directly to Splunk so as to create a dedicated source.

View solution in original post

0 Karma

lguinn2
Legend

Cross-referencing @Sorkin's answer to a timezone question.

0 Karma

vbumgarner
Contributor

I just ran into this problem. 😞

0 Karma

hulahoop
Splunk Employee
Splunk Employee

In the current release of Splunk, in the exact scenario described above, it is not possible to apply TZ when using host/sourcetype overriding. The only alternative is to have the specific hosts forward directly to Splunk so as to create a dedicated source.

0 Karma

jrodman
Splunk Employee
Splunk Employee

You really want to declare the TZ by host, ideally, since the logs are almost certainly generated by some system in either its localtime or in GMT.

However, sometimes life is more complicated than that, like syslog, where the host is identified via a transform, and the original host (used during timestamp extraction) is going to just be where we're acquiring the data.

In this case you're going to just have to use a source pattern to get reasonable behavior, with the hosts split out into files, for example by syslog-ng.

Of course if you have the option of simply altering the timestamp format to include the timezone, that's really ideal for ALL parties, not just Splunk.

jrodman
Splunk Employee
Splunk Employee

If the timezone is declared at time of parsing, it is stored. However, the date parsing code essentially passes the buck for localtime to the system libc, so doesn't know it. Getting the offset is some work. Getting the timezone is pretty hard. However, this sounds like work worth doing. It would be very helpful if you could file an enhancement request with support with one or two use cases to add color to the need.

0 Karma

gfriedmann
Communicator

Man, it would be nice if Splunk's WinEventLog inputs included timezone information.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

The TZ parameter is used and set by the parsing pipeline: http://www.splunk.com/wiki/Where_do_I_configure_my_Splunk_settings%3F and I believe that timestamp processing occurs before any TRANSFORMS. Setting TZ with a host:: or source:: stanza should work (as it does with a sourcetype), so I'm not sure why it isn't working for you. Is there another stanza that might also be matching and overriding?

hulahoop
Splunk Employee
Splunk Employee

Thank you, G. We are still in the eval stage here so are reluctant to make changes like this in the production environment. The preference is to configure Splunk to handle this case if possible. Are you recommending we do not alter the parsing pipeline?

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

oh, okay, yes if you're setting host in props/transforms, then no stanza referring to them is going to work (by that host). Are all events from multiple hosts coming in from that same syslog server? Is it possible to split the hosts from different time zones to different syslog servers or different TCP ports?

0 Karma

hulahoop
Splunk Employee
Splunk Employee

I think that is precisely the problem--"timestamp processing occurs before any TRANSFORMS." This data stream requires a sourcetype and host override. Is there a way to re-process TZ after the sourcetype and host override? Can I create my own processor for this or edit the pipeline order for the parsing queue?

More details: all events are streamed via TCP by a syslog server, except the events are not in syslog format. So we need to add index-time rules to assign sourcetype and host.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...