topic Re: Understanding props and transforms in Splunk Search

Understanding props and transforms

jlixfeld — Thu, 06 Oct 2011 19:36:02 GMT

I'm trying to put into practice what I saw in Michael Wilde's Regex video with regards to making rex searches persistent. I must be missing something because I'm not getting the results I'm after.

(with help from many folks here) I've built a search regex like so. It works perfectly from search:

^(?:\S*\s*){5}(?<hostname>\S*)[^%]*(?<alert>%\S+)

Now, I'm trying to make it persistent by adding those search elements to /local copies of props.conf and transforms.conf.

Since this syslog search is geared mostly towards Cisco syslog output, I see that there is a cisco_syslog stanza in props.conf that I think I can use:

[cisco_syslog]
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
TIME_FORMAT = %b %d %H:%M:%S
TRANSFORMS = syslog-host
REPORT-syslog = syslog-extractions

I copy the entry from default/props.conf and add it to local/props.conf and add the changes:

[cisco_syslog]
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
TIME_FORMAT = %b %d %H:%M:%S
TRANSFORMS = syslog-host
REPORT-syslog = syslog-extractions,syslog-cisco

I add the corresponding stanza to local/transforms.conf:

[syslog-cisco]
REGEX = ^(?:\S*\s*){5}(?<hostname>\S*)[^%]*(?<alert>%\S+)
FORMAT = hostname::$1 alert::$2

Issue the | kv reload=true and change my search sourcetype=syslog, which I'd expect to at least show the new fields in the field picker, but alas, it does not.

Can anyone point me to what I might have missed?

With gratitude,
L:x

Re: Understanding props and transforms

Ayn — Thu, 06 Oct 2011 20:24:07 GMT

When you with extractions using rex or directly in props.conf, you tell Splunk what to call the fields by using the fieldnames within tags in the matching groups. When using the syntax used in transforms.conf, you use the FORMAT directive to achieve that instead. So, you need to change the regex a bit to reflect that, as the tags will now not be interpreted the same way and therefore cause the regex not to match. Remove the tags and you should be fine. So:

[syslog-cisco]
REGEX = ^(?:\S*\s*){5}(\S*)[^%]*(%\S+)
FORMAT = hostname::$1 alert::$2

Re: Understanding props and transforms

jlixfeld — Thu, 06 Oct 2011 21:07:42 GMT

Thanks for the pointer, however that didn't seem to work either.

I modified transforms.conf and removed the tags as you suggested, | kv reload=true, but still I don't see any change in a new search output or in the field picker.

Re: Understanding props and transforms

Ayn — Fri, 07 Oct 2011 07:14:25 GMT

And you say this works with the normal rex command? Curious. What happens if you remove the leading "^"?

Re: Understanding props and transforms

gkanapathy — Fri, 07 Oct 2011 13:56:19 GMT

Quite simply, you applied your new extraction to the sourcetype cisco_syslog, and so it will only be applied to events with that sourcetype. If you search for sourcetype=syslog, you won't get any cisco_syslog events (only syslog), so it won't be applied.

Re: Understanding props and transforms

jlixfeld — Fri, 07 Oct 2011 14:44:26 GMT

Ah, that makes perfect sense. Thank you.

My syslog data input is udp:514, so conceptually speaking, what's the best common practise around using Splunk as a collector from multiple syslog sources (say a Cisco router, a Unix box, a XYZ device)? I imagine it might be quite difficult to have a bunch of transforms on a single syslog stanza to try and account for the differences in many of the log formats, but is it doable? Is it easier to have the local syslog daemon write to different files, let Splunk eat the files and use the different files to apply different formats or transforms?

Re: Understanding props and transforms

jlixfeld — Fri, 07 Oct 2011 17:13:26 GMT

So this is still giving me grief.

I removed the [cisco_syslog] stanza from local/props.conf and replaced it with this:

[syslog]
pulldown_type = true
maxDist = 3
TIME_FORMAT = %b %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 32
TRANSFORMS = syslog-host
REPORT-syslog = syslog-extractions,syslog-cisco
SHOULD_LINEMERGE = False

My local/transforms.conf looks has this as a corresponding entry:

[syslog-cisco]
REGEX = ^(?:\S*\s*){5}(\S*)[^%]*(%\S+)
FORMAT = hostname::$1 alert::$2

Still, on a | kv reload=true or a | extract reload=true followed by a subsequent search for sourcetype=syslog I don't see the new fields in the field picker.

Now I'm going on the assumption that if I have an entry in local/props.conf that also exists in default/props.conf, the local entry will override the default entry, so I'm not running into an issue where it's loading the default entry first and ignoring my local entry?

Re: Understanding props and transforms

jlixfeld — Sat, 08 Oct 2011 15:30:46 GMT

Ok, I solved this. I opted to use props.conf instead of props.conf + transforms.conf. Here's what finally worked:

[source::udp:514]
EXTRACT-name = ^(?:\S*\s*){5}(?<hostname>\S*)[^%]*(?<alert>%\S+)

Thanks again to everyone who chimed in.

Re: Understanding props and transforms

gkanapathy — Sat, 08 Oct 2011 18:46:31 GMT

Oh I just realized the problem in your original. If you have the fields named in the REGEX (e.g., (?<alert>...), then you should not have a FORMAT line. The FORMAT line overwrote your extractions in the transforms.