topic Re: Regex set up in Splunk Search

Regex set up

TheWzrdOz — Thu, 24 May 2012 15:39:01 GMT

Very much a noob here. I've read (or tried to read!) the docs, I've watched the videos and still it's not doing what I thought it should.

I have loaded my data into Splunk (an XML file) and did a rex on the data to find what I was looking for. That worked fine.

sourcetype="xml" | rex "<CommandId>(?<commandid>d+)</CommandId>"

Now I want to persist the data. I added this to my props.conf file (in Splunk\etc\system\local):

[xml]
REPORT-uploads = commandId

And this to transforms.conf (same location):

[commandId]
REGEX=<CommandId>(?<commandId>d+)</CommandId>

Did I do this correctly?

Re: Regex set up

TheWzrdOz — Thu, 24 May 2012 15:39:06 GMT

Markdown is really screwing with the formatting of the text. Not certain how to get around it, either!

Re: Regex set up

TheWzrdOz — Thu, 24 May 2012 16:24:10 GMT

sourcetype="xml" | rex "<CommandId>(?\d+)</CommandId>"

Re: Regex set up

TheWzrdOz — Thu, 24 May 2012 16:24:54 GMT

[xml]

REPORT-uploads = commandId

[commandId]
REGEX = <CommandId>(?\d+)</CommandId>
FORMAT = commandId::$1

Re: Regex set up

lguinn2 — Thu, 24 May 2012 20:19:55 GMT

So first, nicely done so far - but I think you are doing too much work!

With Splunk, you can use the xmlkv command and ask Splunk to do the parsing that you are doing by hand. The only issue is that xml parsing can be slow, so you should search first and then parse. For example, if you are looking for commandid=xyz27, run this search string

sourcetype=xml CommandId xyz27 | 
xmlkv | 
search CommandId=xyz27 | 
whateveryouwant

How this works:

The first line searches for any events that have both the terms "CommandId" and "xyz27". This is not the same as searching for CommandId=xyz27 but it will eliminate a lot of events. This means fewer events to parse in the next step
The xmlkv command will parse the remaining events. Anywhere it sees <tag>value</tag>, it will create a field named tag and set its value to "value". Voila, fields extracted!
The next step actually tests for CommandId=xyz27, eliminating any events that do not match
Finally, you get to do whatever you were ultimately trying to do (statistics, timechart, etc)

If you are going to do this a lot, you might consider saving it as a macro, once you get it sorted out.

Re: Regex set up

lguinn2 — Thu, 24 May 2012 20:32:46 GMT

Second answer:

If you want a direct answer to your question 🙂

I think your regex is probably correct, although I would escape the < characters in the regex (making them \< ) because < by itself has a special meaning to regex.

So if you want to continue with your original solution, try that and forget the xmlkv
But your syntax is a bit wrong, too, I think you need the following in transforms.conf

[commandId]
REGEX=\<CommandId>(d+)\</CommandId>
FORMAT=commandid::$1

Yeah, the markdown gets screwy sometimes when you put in XML or HTML stuff...