Splunk Search

Regex help with Search time field extractions from syslog source

gsawyer1
Engager

I have a string of text from a syslog feed source:

Nov 8 16:16:51 192.168.2.10 Nov 8 16:16:19 SuperServer PES0: Site: Sitename,Server:

Srvr1,Domain: Default,Admin: user1,Policy has been fixed,Policyname - X

I am trying to extract the last three fields from this data, and I can't seem to get it to work. I'm only trying it out in Search first, but am thinking of using an EXTRACT in props.conf finally. The rest of the regex I have doublechecked, using a Perl Regex editor and RegExr, too. All of the syntax seems to be correct, until I try to extract the fields. Here's the regex with search command I have so far:

    Source=syslog | rex ".*,Admin:\s(?<user>\w+),(?<message>.*),(?<policy>.*)" | table user message policy

But my field extraction attempts here yield nothing so far. Any suggestions?

My follow on issue is that I get additional logs from this same source:

Nov 8 16:16:51 192.168.2.10 Nov 8 16:16:19 SuperServer PES0: MACHINE_NAME,Continue,,File Read,Begin: 2012-10-03 01:54:38,End: 2012-10-03 01:54:38,Rule: Log write - media,3196,C:/Windows/System32/svchost.exe,0,No Module Name,C:/Temp/filename.txt,User: SYSTEM,Domain: STANDALONE,Action Type:

Here is the regex I have so far to extract fields from it, too:

Source=syslog | rex ".*PES0:\s(?<machine>\w+),(?<srvr_action_taken>\w+),,(?<user_action_taken>\w+\s\w+),Begin:\.*Rule:\s(?<rule_used>.*),\d+,(?<process_called>.*),\d+,No\sModule\sName,(?<filename>.*),User:\s(?<user>\w+),Domain:\s(?<domain>\w+)" | table _time machine srvr_action_taken user_action_taken rule_used process_called filename user domain

(The table commands are just so I can view the output quickly.)

Also, any suggestions for how I should use these two regexes in a props.conf EXTRACT entry/stanza? I think that's probably the best way to proceed.

Would it make more sense to use a REPORT and have its transform employ DELIM instead?
IS there a way to use both of these regexes against one source, in an EXTRACT or REPORT or transform?

I imagine this will be an easy question for the Splunk Community, but the answer so far escapes me....Any help would be appreciated, I am just starting to learn Regex now....

Tags (2)
0 Karma
1 Solution

emiller42
Motivator

The way you would implement these in your transforms/props.conf is as follows:

In transforms.conf you would actually define two separate transforms.

[foo]
REGEX = .*,Admin:\s(?<user>\w+),(?<message>.*),(?<policy>.*)

[bar]
REGEX = .*PES0:\s(?<machine>\w+),(?<srvr_action_taken>\w+),,(?<user_action_taken>\w+\s\w+),Begin:.*Rule:\s(?<rule_used>.*),\d+,(?<process_called>.*),\d+,No\sModule\sName,(?<filename>.*),User:\s(?<user>\w+),Domain:\s(?<domain>\w+)

Then in your props.conf you reference the above transforms like so:

[syslog]
REPORT-syslog = foo, bar

View solution in original post

emiller42
Motivator

The way you would implement these in your transforms/props.conf is as follows:

In transforms.conf you would actually define two separate transforms.

[foo]
REGEX = .*,Admin:\s(?<user>\w+),(?<message>.*),(?<policy>.*)

[bar]
REGEX = .*PES0:\s(?<machine>\w+),(?<srvr_action_taken>\w+),,(?<user_action_taken>\w+\s\w+),Begin:.*Rule:\s(?<rule_used>.*),\d+,(?<process_called>.*),\d+,No\sModule\sName,(?<filename>.*),User:\s(?<user>\w+),Domain:\s(?<domain>\w+)

Then in your props.conf you reference the above transforms like so:

[syslog]
REPORT-syslog = foo, bar

gsawyer1
Engager

When I used the regex(s) in transforms, I had to put quotes around each comma. Not so when I used the rex command in Search. I also had to change a couple more "\w+" to ".*?" to capture all of the data to their fields.

Either way, thanks to both Rob and emiller42 for their help. It felt good when things started working....
Thanks!! I have a follow-on question I am posting right now, called, "Subsequent transform of an extracted field?" if anyone wants to help....

0 Karma

Rob
Splunk Employee
Splunk Employee

Your rex command might be missing the parameter for the field to extract from. Take a look at the docs here for the rex command:

In this case, try using the following snippet (remove the # symbols due to formatting):

| rex field=_raw ".*,Admin:\s(?<#user>\w+),(?<#message>.*),(?<#policy>.*)" 
0 Karma

Rob
Splunk Employee
Splunk Employee

Also, I forgot to mention that you will want to make sure that field discovery is turned on if you are using Splunk version <5.0 or your fields will not be extracted.

0 Karma

Rob
Splunk Employee
Splunk Employee

The regex you are using for the first example should be ok. The only improvement I could suggest is to make the second field (message) not greedy with (?.*?). With the second regex, it looks like it just needs a few touches (There was an escaped dot in there):

|rex field=_raw .*PES0:\s(?<machine>\w+),(?<srvr_action_taken>\w+),,(?<user_action_taken>\w+\s\w+),Begin:.*?Rule:\s(?<rule_used>.*?),\d+,(?<process_called>.*?),\d+,No\sModule\sName,(?<filename>.*?),User:\s(?<user>\w+),Domain:\s(?<domain>\w+)
0 Karma

gsawyer1
Engager

It would be great if that was the issu, and thanks for the clarification about specifying the field - I've seen so many examples here by now that my head is swimming with them, but I did actually already try specifying field=_raw - but my field count still doesn't increase, and my table command shows the headings I specified, but no data in them, still....

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...