Getting Data In

How to configure both CLEAN_KEYS=false in transforms.conf and KV_MODE=auto in props.conf?

splunkIT
Splunk Employee
Splunk Employee

My logs contain many kv pairs, and some field names contain hyphens characters as well:

timestamp="PST 2015-12-01 11:26:36,400", level="INFO",  x-sid="123456789",  x-ip="127.0.0.1" ,  x-state="ALIVE"

My goal is to retain the hyphen - characters in the field names; thus setting CLEAN_KEYS = false in transforms.conf, and also like splunk to auto-extract the kv pairs:

# props.conf
[mytest]
NO_BINARY_CHECK = true
KV_MODE = auto
REPORT-blah = keephyphen

# transforms.conf
[keephyphen]
CLEAN_KEYS = false

However, despite CLEAN_KEYS = false already set in transforms.conf, splunk still replaces the hyphens in the field names with underscore characters:

x-sid --> x_sid, x-ip --> x_ip , x-state --> x_state

Did I misconfigure the transforms.conf stanza, or does Splunk not support CLEAN_KEYS=false and KV_MODE=auto combination?

0 Karma

rphillips_splk
Splunk Employee
Splunk Employee

Field names must start with a letter and contain only letters, numbers, and underscores. Through testing this it looks like Splunk will not retain (by using CLEAN_KEYS = false) any invalid characters for the field name such as - or / (I see that field names that contain a / are dropped completely where fields containing hyphens are changed to underscore. There is an outstanding enhancement request open (SPL-111920) for such behavior for KV_MODE=auto and CLEAN_KEYS=false to retain the hyphen in the field name. This limitation I'm guessing is because the regex engine doesn't consider those characters to be a valid group structure for the named capture group.

0 Karma

woodcock
Esteemed Legend

MY suspicion is that the mytest specification in props.conf is NOT being triggered at all and that KV_MODE=auto and CLEAN_KEYS=false are working because those are the default values. So make sure that your stuff has source value of mytest from the get-go (not after being overridden) and this should work.

0 Karma

jplumsdaine22
Influencer

It looks ok to me - have you run $SPLUNK_HOME/bin/splunk cmd btool transforms list --debug and $SPLUNK_HOME/bin/splunk cmd btool props list --debug to make sure that the stanzas are being read correctly?

0 Karma

renjith_nair
SplunkTrust
SplunkTrust

Just a quick question, in props, don't you need to mention a spec instead of mytest like source,sourcetype or host just to tell splunk where to apply the transform?

Happy Splunking!
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...