topic Re: How to automatically extract key value pairs after using translatefix in Getting Data In

How to automatically extract key value pairs after using translatefix

isha_rastogi — Tue, 18 Apr 2017 10:24:06 GMT

I have used translatefix to decode the fix messages logs and it worked fine. But Splunk is not able to automatically extract key-value pairs. I don't want to write extract kvdelims and pairdelims in each search. How can I automatically extract key-values using configuration files

My Original data:
19:14:59.338308 outgoing: 8=FIX.4.X|9=12|35=0|34=123456|49=ABC1|52=20170406-23:14:59.338|56=XYZ1|10=123|

After using translatefix as command:
index=abc sourcetype=xyz | translatefix
data is decoded as:

19:14:59.338308 outgoing: BeginString=FIX.4.X BodyLength=12 MsgType=Heartbeat MsgSeqNum=123456 SenderCompID=ABC1 SendingTime=20170406-23:14:59.338 TargetCompID=XYZ1 CheckSum=123

To extract key value pair I have to use extract command in search like below:

ndex="abc" sourcetype=xyz | translatefix | extract pairdelim=" " kvdelim="=" .
Is there any way I can configure to extract automatically using conf files (props.conf, transforms.conf). So I dont have to write extract in each search query and data is extracted automatically.

I tried using delims and regex (\w+)=([^[\s]+) in transforms.conf. It worked for original data but not working after using translatefix command.

Re: How to automatically extract key value pairs after using translatefix

adonio — Tue, 18 Apr 2017 12:37:39 GMT

can you share sample data after it was decoded? (in plain text)

Re: How to automatically extract key value pairs after using translatefix

isha_rastogi — Tue, 18 Apr 2017 13:28:21 GMT

Yes Sure.

My Original data:
19:14:59.338308 outgoing: 8=FIX.4.X|9=12|35=0|34=123456|49=ABC1|52=20170406-23:14:59.338|56=XYZ1|10=123|

After using translatefix as command:
index=abc sourcetype=xyz | translatefix
data is decoded as:

19:14:59.338308 outgoing: BeginString=FIX.4.X BodyLength=12 MsgType=Heartbeat MsgSeqNum=123456 SenderCompID=ABC1 SendingTime=20170406-23:14:59.338 TargetCompID=XYZ1 CheckSum=123

To extract key value pair I have to use extract command in search like below:

ndex="abc" sourcetype=xyz | translatefix | extract pairdelim=" " kvdelim="=" .
Is there any way I can configure to extract automatically using conf files (props.conf, transforms.conf)

Re: How to automatically extract key value pairs after using translatefix

lguinn2 — Tue, 18 Apr 2017 15:38:19 GMT

In the props.conf (on the search head or wherever users log in), add the following:

[xyz]
KV_MODE = auto

And the key-value pairs (separated by 😃 will be extracted. You do not need the transforms.conf.

Re: How to automatically extract key value pairs after using translatefix

isha_rastogi — Wed, 19 Apr 2017 06:20:26 GMT

Tried to put KV_MODE =auto. No success :(. ( Tried on new indexed data)

Re: How to automatically extract key value pairs after using translatefix

lguinn2 — Wed, 19 Apr 2017 20:19:03 GMT

Okay, I gave a dumb answer because I didn't realize that you are using translatefix inline. If you are reformatting the data on the fly (which is how translatefix works), you can't put the field extractions in props.conf or transforms.conf based on the NEW format.

However, you can certainly add the field extractions based on the original/stored format. Of course, in the original format, you don't get the "real" field names, instead you get weird names like "10" and "56." To make the names pretty, you would need to do a bunch of renames, which doesn't really help either.

I suggest a macro. Let's call it infixed_xyz() and let the body of the macro be

sourcetype=xyz | translatefix | extract pairdelim=" " kvdelim="="

now you can write a search like this

index=abc `infixed_xyx` | where TargetCompID=XYZ1 | stats or whatever

Re: How to automatically extract key value pairs after using translatefix

isha_rastogi — Thu, 20 Apr 2017 15:50:19 GMT

As translatefix is configured in commands.conf and I'm using it as command. Can I configure it in configuration file so that it will be automatically applied instead of writing it inline.