Getting Data In

How to apply a timezone (TZ) props.conf entry to a rewritten source or host?

eddit0r
Explorer

I have a datasource which includes a timestamp without a timezone forwarded from a single collection source, and some other location data in the event which I can use to categorise the data into their respective timezones.

As the data is forwarded from a single source, I cannot set the timezones at input.confs without a script to pre-process the data, which I would like to avoid. The output format of the data is fixed and I have no access to the source.

Source: foo
Sourcetype: bar
Event: AAA,foobar,AUVI,201412121738,324521345,3452345,IVXIJB1,3,0,1,2346,357,8

So we have the following props and transforms:

props.conf
[bar]
SHOULD_LINEMERGE = false
TIME_PREFIX = (?:[^,]*,){3}
TIME_FORMAT = %Y%m%d%H%M
TRANSFORMS-bar_create_host = bar_create_host

[host::AUVI*]
TZ = Australia/Melbourne

[host::AUNS*]
TZ = Australia/Sydney

[host::AUQL*]
TZ = Australia/Brisbane

[host::AUSA*]
TZ = Australia/Adelaide

transforms.conf
[bar_create_host]
SOURCE_KEY = _raw
REGEX = (?^AAA),(?[a-zA-Z0-9]*),(?[A-Z0-9]*)
FORMAT = $3
DEST_KEY = MetaData:Host`

The host gets rewritten fine, the timestamp extracts fine, but the timezone will not apply based on the rewritten host stanza in props.conf. the date_zone field is always 'local' (should be 600 if set to Australia/Melbourne / UTC+10:00)

I have tried rewriting the source instead of the host field and setting the TZ entry under a source::foo based stanza in props.conf, I have tried setting priority of the associated props stanzas to 1 and 100 in alternating configs to no avail.

At the moment I'm stuck on using a heavy forwarder to do the host rewrite then forcing the input of the cooked data to be reparsed at the indexer so the TZ entry is honoured on a per-host or per-source basis. That is really more cumbersome than I would like.

Is this a bug or is the parsing of these events working as designed, that is you cannot set the TZ at index/parsing time by a rewritten host or source stanza?

Any assistance appreciated. I would like to avoid a scripted input if possible.

This is in a distributed install on RHEL/OEL 5 & 6 x64, Splunk version 6.1.5 on forwarders, indexers and search heads.

0 Karma

woodcock
Esteemed Legend

As you noted, the problem is that you have overridden your host and you cannot use the new host value as a stanza header in props.conf; therefore, you need to trigger off of source or sourcetype and you may have more options than you suppose. The easiest option is to write your files into a directory that directly (/mydir/TZ-Central/) or indirectly (/mydir/hostx/) identifies something that you can use later with a source-based stanza header like this:

[source::/mydir/TZ-Central/*]
TZ=US/Central
[source::/mydir/hostx/*]
TZ=US/Eastern
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...