Getting Data In
Highlighted

How to configure both CLEAN_KEYS=false in transforms.conf and KV_MODE=auto in props.conf?

Splunk Employee
Splunk Employee

My logs contain many kv pairs, and some field names contain hyphens characters as well:

timestamp="PST 2015-12-01 11:26:36,400", level="INFO",  x-sid="123456789",  x-ip="127.0.0.1" ,  x-state="ALIVE"

My goal is to retain the hyphen - characters in the field names; thus setting CLEAN_KEYS = false in transforms.conf, and also like splunk to auto-extract the kv pairs:

# props.conf
[mytest]
NO_BINARY_CHECK = true
KV_MODE = auto
REPORT-blah = keephyphen

# transforms.conf
[keephyphen]
CLEAN_KEYS = false

However, despite CLEAN_KEYS = false already set in transforms.conf, splunk still replaces the hyphens in the field names with underscore characters:

x-sid --> xsid, x-ip --> xip , x-state --> x_state

Did I misconfigure the transforms.conf stanza, or does Splunk not support CLEANKEYS=false and KVMODE=auto combination?

0 Karma
Highlighted

Re: How to configure both CLEAN_KEYS=false in transforms.conf and KV_MODE=auto in props.conf?

SplunkTrust
SplunkTrust

Just a quick question, in props, don't you need to mention a spec instead of mytest like source,sourcetype or host just to tell splunk where to apply the transform?

0 Karma
Highlighted

Re: How to configure both CLEAN_KEYS=false in transforms.conf and KV_MODE=auto in props.conf?

Influencer

It looks ok to me - have you run $SPLUNK_HOME/bin/splunk cmd btool transforms list --debug and $SPLUNK_HOME/bin/splunk cmd btool props list --debug to make sure that the stanzas are being read correctly?

0 Karma
Highlighted

Re: How to configure both CLEAN_KEYS=false in transforms.conf and KV_MODE=auto in props.conf?

Esteemed Legend

MY suspicion is that the mytest specification in props.conf is NOT being triggered at all and that KV_MODE=auto and CLEAN_KEYS=false are working because those are the default values. So make sure that your stuff has source value of mytest from the get-go (not after being overridden) and this should work.

0 Karma
Highlighted

Re: How to configure both CLEAN_KEYS=false in transforms.conf and KV_MODE=auto in props.conf?

Splunk Employee
Splunk Employee

Field names must start with a letter and contain only letters, numbers, and underscores. Through testing this it looks like Splunk will not retain (by using CLEANKEYS = false) any invalid characters for the field name such as - or / (I see that field names that contain a / are dropped completely where fields containing hyphens are changed to underscore. There is an outstanding enhancement request open (SPL-111920) for such behavior for KVMODE=auto and CLEAN_KEYS=false to retain the hyphen in the field name. This limitation I'm guessing is because the regex engine doesn't consider those characters to be a valid group structure for the named capture group.

0 Karma