topic Re: Field Extractions in Splunk Search

Field Extractions

Hazel — Wed, 21 Apr 2010 21:22:25 GMT

Hello,

I am trying to configure a props/transforms and it is not working. it does not come up as an extra field that i can select on the web search.

Can you see anything wrong with this?

The path i am extracting from looks something like /app/ems/envs/U1_LN_DERIV_TMS/logs/trace.log

I want to pick up the directory name in the middle - U1_LN_DERIV_TMS.

I even tried changing my regex to (.*) to see if there was a problem with the regex (which i have tested in the splunk web and worked fine) and it still doesn't come through.

Have i missed something obvious? Have also tried it with and without the quotes around the REGEX

props.conf:

[source::///app/ems/envs/uat/.../logs/trace.log]
REPORT-ems = instanceName

transforms.conf

[instanceName]
SOURCE_KEY = source
REGEX=\/app\/ems\/envs\/uat\/([A-Z0-9_]*)\/logs\/trace.log
FORMAT = emsinstance::$1

Re: Field Extractions

gkanapathy — Wed, 21 Apr 2010 21:50:24 GMT

Too many slashes at the beginning of [source:///...]. The extra slashes are only used (rather unnecessarily anyway) in inputs.conf stanzas, not props.conf.

Also, you can test regexes at the CLI directly using the | rex search command:

... | rex field=source "/app/ems/envs/uat/(?<emsinstance>\w*)/logs/trace.log"

Re: Field Extractions

Hazel — Wed, 21 Apr 2010 22:08:26 GMT

Thanks I will try this - I have already been using rex to test my regex which is why i assumed it was probably a syntax issue rather than a regex issue!

Re: Field Extractions

Lowell — Thu, 29 Apr 2010 02:00:10 GMT

gkanapathy is right about your problem is. I just had a few thoughts that wouldn't fit in a comment.

Just so you know, you can use named capture groups, like (?<emsinstance>\w+) in your transforms.conf file just like you do using rex. When your using named groups, you don't need the FORMAT line anymore. I found that this approach makes it quicker to jump between interactive testing with rex and then copying the regex into my transforms.conf file.

In this case, you also have the option of using an EXTRACT entry in your props.conf file. I rather prefer this approach, since you can setup everything in one place, and you don't to come up with some arbitrary name transformer name. 😉

Here's an example in props.conf:

[source::/app/ems/envs/uat/*/logs/trace.log]
EXTRACT-fields = [/\\]uat[/\\](?[\w_]+)[/\\]logs[/\\] in source

Notes: I use [/\\] because it will match both unix-style and dos-style paths (if that's a concern for you and you can use that in your source config entry too, although it looks ugly: [source::[/\\]app[/\\]ems[/\\]envs[/\\]uat[/\\]*[/\\]logs[/\\]trace.log]). You may also note that I'm not matching the whole path in the regex, which is because in this case, you really don't need to, your [source::...] stanza is doing that work for you already.