Transforms.conf and props.conf field extractions

[Michael_Schyma1]

Hey fellow Splunker's. I am trying to figure out what i am doing wrong in the transforms.conf to create the proper field extractions. I feel like this should work. The format of my regex should be right, but i am not getting any results returned in the fields that I want to extract. Any help that can be given to fix this issue would be greatly appreachated. Thank you.

Below is an example of the transforms.conf and the props.conf files.

(Transforms.conf)

[vmware-host]
REGEX = ^(?:\S+\s){3}(?P<Hostname>\S+)(?:\S+\s){4}(?P<ANSI_Standard>[^:]+)(?:\S+\s){3}(?P<Message_ID>\S+)\s(?P<Priority>\S+)\s\'(?P<Category>\w+)\'\]\s\[(?P<Event_Type>[^\]]+)\]
FORMAT = fieldOne::$1, fieldTwo::$2

(props.conf)

[VMwareESXi]
pulldown_type = 1
TRANSFORMS = vmware-host

[lguinn2 · Accepted solution]

Thanks for the updates. Now I can see a reason to use the transforms. But, as Bob pointed out in the comment below, you should use REPORT instead of TRANSFORM in props.conf. It is much better to build a search-time field extraction.

props.conf

[VMwareESXi]
pulldown_type = 1
REPORT-r1 = vmware-host

transforms.conf

[vmware-host]
REGEX=(\w+\s\d+\s\d+:\d+:\d+)\s(\S+)\s(\w+\s\d+\s\d+:\d+:\d+)\s(\S+)\[(\S+\s\S+)\s(\S+)\s(\S+)\'(\S+)\'\] \s*(.*)
FORMAT=date1::$1,hostname::$2, date2::$3, servicename::$4, date3::$5,hexcode::$6,reportinglevel::$7, reportingcomponent::$8,message::$9

Note that you don't want to break the lines. So if you copy-and-paste this (which is a good idea), put the lines for the REGEX and the FORMAT back together again!
I named your fields somewhat arbitrarily, but hopefully this will give you a good idea of how to finish the job.

[lguinn2]

Bob - you are so right! I must have been brain-dead on this one! I have updated my answer to reflect your comment.

[BobM]

This should work but it is not a best practice as it extracts the field at index time. This places more load on your indexers as well as taking more space in your index. If you replace TRANSFORMS with REPORT in your props.conf this will run at search time which is more efficient.

Although switching to EXTRACT would be my preference. Your initial post was close to an extract regex.

[lguinn2]

I am not sure why you need the transforms.conf stanza at all, given the way you wrote your regular expression.

Why not

[VMwareESXi]
pulldown_type = 1
EXTRACT-e1 =^(?:\S+\s){3}(?P<Hostname>\S+)(?:\S+\s){4}(?P<ANSI_Standard>[^:]+)(?:\S+\s){3}(?P<Message_ID>\S+)\s(?P<Priority>\S+)\s\'(?P<Category>\w+)\'\]\s\[(?P<Event_Type>[^\]]+)\]

But I do see some strange things in the regular expression. Can you post a snippet of the ESXi log file that you are trying to use?

[Michael_Schyma1]

If any additional information or logs are needed, just let me know. I will be keeping a close watch on this question.

[Michael_Schyma1]

Thank you so much.
Via the ESXi sourcetype:

Sep 5 14:21:34 cam.company.com Sep 5 18:21:34 Vpxa: [2012-09-05 18:21:34.420 17C3DB90 verbose 'App'] Set internal stats for VM: 272 (vpxa VM id), 4181 (vpxd VM id). Is FT primary? 0
Sep 5 14:21:34 cam.company.com Sep 5 18:21:34 Vpxa: [2012-09-05 18:21:34.664 17B7AB90 verbose 'VpxaHalCnxHostagent'] Received callback in WaitForUpdatesDone
Sep 5 14:21:34 cam.company.com Sep 5 18:21:34 Vpxa: [2012-09-05 18:21:34.664 17B7AB90 verbose 'VpxaHalCnxHostagent'] [VpxaHalCnxHostagent::ProcessUpdate] Applying updates from 3675467 to 3675468 (at 3675467)

[Michael_Schyma1]

I removed the parts of the configuration files that were not necessary.

[landen99]

Keep in mind that every set of parenthesis is a capture group. $1 goes to the very first capture group, not the first named capture group.

[Michael_Schyma1]

Thank you for responding. We are just commenting them out to try and figure out what the issue is of why we can not get the regex to work. We are trying to get the transforms.conf file to work the [vmware-host]

[Ayn]

Also, which of the settings are you referring to? I see multiple things going on, of which some parts are commented out. What are you expecting, and what's not working, specifically?

[Ayn]

Please indent code sections with 4 spaces at the beginning of the line. That way they will show correctly here on the site. Thank you.