How to apply a timezone (TZ) props.conf entry to a rewritten source or host?

eddit0r — Fri, 12 Dec 2014 07:07:24 GMT

I have a datasource which includes a timestamp without a timezone forwarded from a single collection source, and some other location data in the event which I can use to categorise the data into their respective timezones.

As the data is forwarded from a single source, I cannot set the timezones at input.confs without a script to pre-process the data, which I would like to avoid. The output format of the data is fixed and I have no access to the source.

Source: foo
Sourcetype: bar
Event: AAA,foobar,AUVI,201412121738,324521345,3452345,IVXIJB1,3,0,1,2346,357,8

So we have the following props and transforms:

props.conf
[bar]
SHOULD_LINEMERGE = false
TIME_PREFIX = (?:[^,]*,){3}
TIME_FORMAT = %Y%m%d%H%M
TRANSFORMS-bar_create_host = bar_create_host

[host::AUVI*]
TZ = Australia/Melbourne

[host::AUNS*]
TZ = Australia/Sydney

[host::AUQL*]
TZ = Australia/Brisbane

[host::AUSA*]
TZ = Australia/Adelaide

transforms.conf
[bar_create_host]
SOURCE_KEY = _raw
REGEX = (?^AAA),(?[a-zA-Z0-9]*),(?[A-Z0-9]*)
FORMAT = $3
DEST_KEY = MetaData:Host`

The host gets rewritten fine, the timestamp extracts fine, but the timezone will not apply based on the rewritten host stanza in props.conf. the date_zone field is always 'local' (should be 600 if set to Australia/Melbourne / UTC+10:00)

I have tried rewriting the source instead of the host field and setting the TZ entry under a source::foo based stanza in props.conf, I have tried setting priority of the associated props stanzas to 1 and 100 in alternating configs to no avail.

At the moment I'm stuck on using a heavy forwarder to do the host rewrite then forcing the input of the cooked data to be reparsed at the indexer so the TZ entry is honoured on a per-host or per-source basis. That is really more cumbersome than I would like.

Is this a bug or is the parsing of these events working as designed, that is you cannot set the TZ at index/parsing time by a rewritten host or source stanza?

Any assistance appreciated. I would like to avoid a scripted input if possible.

This is in a distributed install on RHEL/OEL 5 & 6 x64, Splunk version 6.1.5 on forwarders, indexers and search heads.

Re: How to apply a timezone (TZ) props.conf entry to a rewritten source or host?

woodcock — Mon, 29 Jun 2015 04:01:40 GMT

As you noted, the problem is that you have overridden your host and you cannot use the new host value as a stanza header in props.conf; therefore, you need to trigger off of source or sourcetype and you may have more options than you suppose. The easiest option is to write your files into a directory that directly (/mydir/TZ-Central/) or indirectly (/mydir/hostx/) identifies something that you can use later with a source-based stanza header like this:

[source::/mydir/TZ-Central/*]
TZ=US/Central
[source::/mydir/hostx/*]
TZ=US/Eastern

topic How to apply a timezone (TZ) props.conf entry to a rewritten source or host? in Getting Data In

How to apply a timezone (TZ) props.conf entry to a rewritten source or host?

Re: How to apply a timezone (TZ) props.conf entry to a rewritten source or host?