topic Setting MetaData:Host over transforms.conf doesn't work in Splunk Search

Setting MetaData:Host over transforms.conf doesn't work

Thomas_Gresch — Tue, 14 Sep 2010 19:57:41 GMT

I have icinga debug logs from a server called monitoring01 looking like:

[1284468200.195107] Checking service 'sys - Zeus ZXTM LB zeus.flipper processes' on host 'balance01'...

monitoring01 is a splunk forwarder. Now I want to rename the host bit on splunk from monitoring01 to whatever host is mentioned in the logfile, in the above example that would be 'balance01'.

On monitoring01 (splunk forwarder) I have the following files in place. They should convert the time and the hostname:

/opt/splunk/etc/apps/scripts/props.conf:

[script://./bin/icinga_converter.sh]
TIME_PREFIX = \[\d{10}
TIME_FORMAT = %+
MAX_TIMESTAMP_LOOKAHEAD = 11
SHOULD_LINEMERGE = false
TRANSFORMS-hostname = icinga_hostconverter

/opt/splunk/etc/apps/scripts/transforms.conf:

[icinga_hostconverter]
REGEX = ([^']*)'\.\.\.$
FORMAT = host::$1
DEST_KEY = MetaData:Host

The timestamp is taken out of the logline instead of arrival time at splunk correctly, but MetaData:Host remains to be set as monitoring01.

I can't find any hint, why the transformation won't work. Does anybody have an idea?

Re: Setting MetaData:Host over transforms.conf doesn't work

Jeremiah — Wed, 15 Sep 2010 01:00:09 GMT

Are the paths correct? transforms.conf and props.conf should go into either a default or local directory in your application (../etc/apps/scripts/default/transforms.conf).

Re: Setting MetaData:Host over transforms.conf doesn't work

hulahoop — Wed, 15 Sep 2010 01:57:26 GMT

Thomas, are you using a regular forwarder or a lightweight forwarder? If you are using a LWF, then your host transform will not be honored. If this is the case, then you should put your host extraction configuration on the indexer.

Re: Setting MetaData:Host over transforms.conf doesn't work

Thomas_Gresch — Sun, 19 Sep 2010 20:05:04 GMT

I've tried moving them into a local/ and a default/ directory within the app - no effect.

Re: Setting MetaData:Host over transforms.conf doesn't work

Thomas_Gresch — Sun, 19 Sep 2010 20:21:00 GMT

I've switched the forwarder from a LightWeight forwarder to a regular forwarder:

'splunk display app' shows

SplunkForwarder UNCONFIGURED ENABLED INVISIBLE

SplunkLightForwarder UNCONFIGURED DISABLED INVISIBLE

but still no effect.

Re: Setting MetaData:Host over transforms.conf doesn't work

tskimball — Wed, 02 Mar 2011 17:50:10 GMT

You have your source in props.conf as type script:: - Are you doing an internal pull using this script?

Try doing a plain forwarding of the raw file to the indexer, and specify source:: at the indexing props.conf instead.