topic Re: Transforms.conf and props.conf field extractions in Splunk Search

Transforms.conf and props.conf field extractions

Michael_Schyma1 — Wed, 05 Sep 2012 15:52:59 GMT

Hey fellow Splunker's. I am trying to figure out what i am doing wrong in the transforms.conf to create the proper field extractions. I feel like this should work. The format of my regex should be right, but i am not getting any results returned in the fields that I want to extract. Any help that can be given to fix this issue would be greatly appreachated. Thank you.

Below is an example of the transforms.conf and the props.conf files.

(Transforms.conf)

[vmware-host]
REGEX = ^(?:\S+\s){3}(?P<Hostname>\S+)(?:\S+\s){4}(?P<ANSI_Standard>[^:]+)(?:\S+\s){3}(?P<Message_ID>\S+)\s(?P<Priority>\S+)\s\'(?P<Category>\w+)\'\]\s\[(?P<Event_Type>[^\]]+)\]
FORMAT = fieldOne::$1, fieldTwo::$2

(props.conf)

[VMwareESXi]
pulldown_type = 1
TRANSFORMS = vmware-host

Re: Transforms.conf and props.conf field extractions

Ayn — Wed, 05 Sep 2012 16:16:57 GMT

Please indent code sections with 4 spaces at the beginning of the line. That way they will show correctly here on the site. Thank you.

Re: Transforms.conf and props.conf field extractions

Ayn — Wed, 05 Sep 2012 16:18:06 GMT

Also, which of the settings are you referring to? I see multiple things going on, of which some parts are commented out. What are you expecting, and what's not working, specifically?

Re: Transforms.conf and props.conf field extractions

Michael_Schyma1 — Wed, 05 Sep 2012 16:53:14 GMT

Thank you for responding. We are just commenting them out to try and figure out what the issue is of why we can not get the regex to work. We are trying to get the transforms.conf file to work the [vmware-host]

Re: Transforms.conf and props.conf field extractions

Michael_Schyma1 — Wed, 05 Sep 2012 17:23:35 GMT

I removed the parts of the configuration files that were not necessary.

Re: Transforms.conf and props.conf field extractions

lguinn2 — Wed, 05 Sep 2012 18:13:12 GMT

I am not sure why you need the transforms.conf stanza at all, given the way you wrote your regular expression.

Why not

[VMwareESXi]
pulldown_type = 1
EXTRACT-e1 =^(?:\S+\s){3}(?P<Hostname>\S+)(?:\S+\s){4}(?P<ANSI_Standard>[^:]+)(?:\S+\s){3}(?P<Message_ID>\S+)\s(?P<Priority>\S+)\s\'(?P<Category>\w+)\'\]\s\[(?P<Event_Type>[^\]]+)\]

But I do see some strange things in the regular expression. Can you post a snippet of the ESXi log file that you are trying to use?

Re: Transforms.conf and props.conf field extractions

Michael_Schyma1 — Wed, 05 Sep 2012 18:28:09 GMT

Thank you so much.
Via the ESXi sourcetype:

Sep 5 14:21:34 cam.company.com Sep 5 18:21:34 Vpxa: [2012-09-05 18:21:34.420 17C3DB90 verbose 'App'] Set internal stats for VM: 272 (vpxa VM id), 4181 (vpxd VM id). Is FT primary? 0
Sep 5 14:21:34 cam.company.com Sep 5 18:21:34 Vpxa: [2012-09-05 18:21:34.664 17B7AB90 verbose 'VpxaHalCnxHostagent'] Received callback in WaitForUpdatesDone
Sep 5 14:21:34 cam.company.com Sep 5 18:21:34 Vpxa: [2012-09-05 18:21:34.664 17B7AB90 verbose 'VpxaHalCnxHostagent'] [VpxaHalCnxHostagent::ProcessUpdate] Applying updates from 3675467 to 3675468 (at 3675467)

Re: Transforms.conf and props.conf field extractions

Michael_Schyma1 — Wed, 05 Sep 2012 18:43:25 GMT

If any additional information or logs are needed, just let me know. I will be keeping a close watch on this question.

Re: Transforms.conf and props.conf field extractions

lguinn2 — Thu, 06 Sep 2012 22:51:08 GMT

Thanks for the updates. Now I can see a reason to use the transforms. But, as Bob pointed out in the comment below, you should use REPORT instead of TRANSFORM in props.conf. It is much better to build a search-time field extraction.

props.conf

[VMwareESXi]
pulldown_type = 1
REPORT-r1 = vmware-host

transforms.conf

[vmware-host]
REGEX=(\w+\s\d+\s\d+:\d+:\d+)\s(\S+)\s(\w+\s\d+\s\d+:\d+:\d+)\s(\S+)\[(\S+\s\S+)\s(\S+)\s(\S+)\'(\S+)\'\] \s*(.*)
FORMAT=date1::$1,hostname::$2, date2::$3, servicename::$4, date3::$5,hexcode::$6,reportinglevel::$7, reportingcomponent::$8,message::$9

Note that you don't want to break the lines. So if you copy-and-paste this (which is a good idea), put the lines for the REGEX and the FORMAT back together again!
I named your fields somewhat arbitrarily, but hopefully this will give you a good idea of how to finish the job.

Re: Transforms.conf and props.conf field extractions

BobM — Wed, 12 Sep 2012 13:36:42 GMT

This should work but it is not a best practice as it extracts the field at index time. This places more load on your indexers as well as taking more space in your index. If you replace TRANSFORMS with REPORT in your props.conf this will run at search time which is more efficient.

Although switching to EXTRACT would be my preference. Your initial post was close to an extract regex.

Re: Transforms.conf and props.conf field extractions

lguinn2 — Thu, 13 Sep 2012 08:22:27 GMT

Bob - you are so right! I must have been brain-dead on this one! I have updated my answer to reflect your comment.

Re: Transforms.conf and props.conf field extractions

landen99 — Tue, 08 Sep 2015 17:18:46 GMT

Keep in mind that every set of parenthesis is a capture group. $1 goes to the very first capture group, not the first named capture group.