topic Regex help with Search time field extractions from syslog source in Splunk Search

Regex help with Search time field extractions from syslog source

gsawyer1 — Wed, 07 Nov 2012 21:04:50 GMT

I have a string of text from a syslog feed source:

Nov 8 16:16:51 192.168.2.10 Nov 8 16:16:19 SuperServer PES0: Site: Sitename,Server:

Srvr1,Domain: Default,Admin: user1,Policy has been fixed,Policyname - X

I am trying to extract the last three fields from this data, and I can't seem to get it to work. I'm only trying it out in Search first, but am thinking of using an EXTRACT in props.conf finally. The rest of the regex I have doublechecked, using a Perl Regex editor and RegExr, too. All of the syntax seems to be correct, until I try to extract the fields. Here's the regex with search command I have so far:

    Source=syslog | rex ".*,Admin:\s(?<user>\w+),(?<message>.*),(?<policy>.*)" | table user message policy

But my field extraction attempts here yield nothing so far. Any suggestions?

My follow on issue is that I get additional logs from this same source:

Nov 8 16:16:51 192.168.2.10 Nov 8 16:16:19 SuperServer PES0: MACHINE_NAME,Continue,,File Read,Begin: 2012-10-03 01:54:38,End: 2012-10-03 01:54:38,Rule: Log write - media,3196,C:/Windows/System32/svchost.exe,0,No Module Name,C:/Temp/filename.txt,User: SYSTEM,Domain: STANDALONE,Action Type:

Here is the regex I have so far to extract fields from it, too:

Source=syslog | rex ".*PES0:\s(?<machine>\w+),(?<srvr_action_taken>\w+),,(?<user_action_taken>\w+\s\w+),Begin:\.*Rule:\s(?<rule_used>.*),\d+,(?<process_called>.*),\d+,No\sModule\sName,(?<filename>.*),User:\s(?<user>\w+),Domain:\s(?<domain>\w+)" | table _time machine srvr_action_taken user_action_taken rule_used process_called filename user domain

(The table commands are just so I can view the output quickly.)

Also, any suggestions for how I should use these two regexes in a props.conf EXTRACT entry/stanza? I think that's probably the best way to proceed.

Would it make more sense to use a REPORT and have its transform employ DELIM instead?
IS there a way to use both of these regexes against one source, in an EXTRACT or REPORT or transform?

I imagine this will be an easy question for the Splunk Community, but the answer so far escapes me....Any help would be appreciated, I am just starting to learn Regex now....

Re: Regex help with Search time field extractions from syslog source

Rob — Wed, 07 Nov 2012 21:15:24 GMT

Your rex command might be missing the parameter for the field to extract from. Take a look at the docs here for the rex command:

In this case, try using the following snippet (remove the # symbols due to formatting):

| rex field=_raw ".*,Admin:\s(?<#user>\w+),(?<#message>.*),(?<#policy>.*)"

Re: Regex help with Search time field extractions from syslog source

gsawyer1 — Wed, 07 Nov 2012 21:23:38 GMT

It would be great if that was the issu, and thanks for the clarification about specifying the field - I've seen so many examples here by now that my head is swimming with them, but I did actually already try specifying field=_raw - but my field count still doesn't increase, and my table command shows the headings I specified, but no data in them, still....

Re: Regex help with Search time field extractions from syslog source

Rob — Wed, 07 Nov 2012 22:11:26 GMT

The regex you are using for the first example should be ok. The only improvement I could suggest is to make the second field (message) not greedy with (?.*?). With the second regex, it looks like it just needs a few touches (There was an escaped dot in there):

|rex field=_raw .*PES0:\s(?<machine>\w+),(?<srvr_action_taken>\w+),,(?<user_action_taken>\w+\s\w+),Begin:.*?Rule:\s(?<rule_used>.*?),\d+,(?<process_called>.*?),\d+,No\sModule\sName,(?<filename>.*?),User:\s(?<user>\w+),Domain:\s(?<domain>\w+)

Re: Regex help with Search time field extractions from syslog source

emiller42 — Wed, 07 Nov 2012 22:37:45 GMT

The way you would implement these in your transforms/props.conf is as follows:

In transforms.conf you would actually define two separate transforms.

[foo]
REGEX = .*,Admin:\s(?<user>\w+),(?<message>.*),(?<policy>.*)

[bar]
REGEX = .*PES0:\s(?<machine>\w+),(?<srvr_action_taken>\w+),,(?<user_action_taken>\w+\s\w+),Begin:.*Rule:\s(?<rule_used>.*),\d+,(?<process_called>.*),\d+,No\sModule\sName,(?<filename>.*),User:\s(?<user>\w+),Domain:\s(?<domain>\w+)

Then in your props.conf you reference the above transforms like so:

[syslog]
REPORT-syslog = foo, bar

Re: Regex help with Search time field extractions from syslog source

Rob — Wed, 07 Nov 2012 22:40:43 GMT

Also, I forgot to mention that you will want to make sure that field discovery is turned on if you are using Splunk version <5.0 or your fields will not be extracted.

Re: Regex help with Search time field extractions from syslog source

gsawyer1 — Fri, 09 Nov 2012 15:14:28 GMT

When I used the regex(s) in transforms, I had to put quotes around each comma. Not so when I used the rex command in Search. I also had to change a couple more "\w+" to ".*?" to capture all of the data to their fields.

Either way, thanks to both Rob and emiller42 for their help. It felt good when things started working....
Thanks!! I have a follow-on question I am posting right now, called, "Subsequent transform of an extracted field?" if anyone wants to help....