topic Re: How can I do field extraction on a security log using regex? in Splunk Search

How can I do field extraction on a security log using regex?

Log_wrangler — Wed, 14 Feb 2018 22:48:31 GMT

I need a little guidance on rex field extraction on the following "redacted" security log. Unfortunately, I don't have permission to use field extraction gui so need to rex in the search.

2018:02:09-23:04:30 blahblah httpd[11111], : [foo-security:error],  [pid 12345:tid 1234567890],  [client 192.168.1.1],  Foo-Security: Warning. Pattern match "some expression..." at ARGS:file. [file "some_attack.conf"],  [line "111"],  [id "111"],  [rev "1"],  [msg "Some Attack: blah blah Detected."],  [data "Matched Data: blah blah ARGS:file: <foo=bar"],  [severity "CRITICAL"],  [ver "foo/2.2. [hostname "bar.com"],  [uri "/foo/bar/foobar"],  [unique_id "a1b2c3-a1b2c3"],

As you can see, there are commas and [ brackets] delimiting the log and splunk is extracting some fields like the time field correctly, but I would like to extract more fields

like this

2018:02:09-23:04:30 
blahblah httpd[11111], : <-----------field 1
[foo-security:error],  <-----------field 2 
[pid 12345:tid 1234567890],  <-----------field 3
[client 192.168.1.1],  <-----------field 4
Foo-Security: Warning. Pattern match "some expression..." at ARGS:file. <-----------field 5
[file "some_attack.conf"],  <-----------field 6
[line "111"],  <-----------field 7
[id "111"],  <-----------field 8
[rev "1"],  <-----------field 9
[msg "Some Attack: blah blah Detected."],  <-----------field 10
[data "Matched Data: blah blah ARGS:file: <foo=bar"],  <-----------field 11
[severity "CRITICAL"],  <-----------field 12
[ver "foo/2.2. [hostname "bar.com"],  <-----------field 13
[uri "/foo/bar/foobar"],  <-----------field 14
[unique_id "a1b2c3-a1b2c3"], <-----------field 15

I am a bit rusty on this so any help is much appreciated.

Thank you

Re: How can I do field extraction on a security log using regex?

Log_wrangler — Thu, 15 Feb 2018 02:09:32 GMT

for msg field number 10 would the following spl code be correct?

... | regex _raw="msg+.[^],]*

Re: How can I do field extraction on a security log using regex?

493669 — Thu, 15 Feb 2018 04:02:39 GMT

Hi,
Try like this run anywhere search:

|makeresults|eval _raw="2018:02:09-23:04:30 blahblah httpd[11111], : [foo-security:error],  [pid 12345:tid 1234567890],  [client 192.168.1.1],  Foo-Security: Warning. Pattern match \"some expression...\" at ARGS:file. [file \"some_attack.conf\"],  [line \"111\"],  [id \"111\"],  [rev \"1\"],  [msg \"Some Attack: blah blah Detected.\"],  [data \"Matched Data: blah blah ARGS:file: <foo=bar\"],  [severity \"CRITICAL\"],  [ver \"foo/2.2. [hostname \"bar.com\"],  [uri \"/foo/bar/foobar\"],  [unique_id \"a1b2c3-a1b2c3\"], "|rex "^[\d\:]+-[\d\:]+(?<_raw>.*)"|makemv _raw delim=","|eval field0=mvindex(_raw,0),field1=mvindex(_raw,1),field2=mvindex(_raw,2),field3=mvindex(_raw,3),field4=mvindex(_raw,4),field5=mvindex(_raw,5),field6=mvindex(_raw,6),field7=mvindex(_raw,7)

Re: How can I do field extraction on a security log using regex?

Log_wrangler — Thu, 15 Feb 2018 14:52:39 GMT

Thank you for the reply. Maybe I am misunderstanding or I did not explain clearly.

I was looking for a way that will eval all the log events in _raw, without having to copy and paste like above.

The example I provided is the un-parsed / un-extracted raw text I need to rex into usable fields. Does that make sense...?

Re: How can I do field extraction on a security log using regex?

Log_wrangler — Thu, 15 Feb 2018 14:56:23 GMT

the logs are in an index=A and sourcetype = A, etc... is there a way to use your method with a specific index and sourcetype?

Re: How can I do field extraction on a security log using regex?

493669 — Thu, 15 Feb 2018 14:59:50 GMT

if the event is already indexed then have you tried this:

<base_search>|makemv _raw delim=","|eval field0=mvindex(_raw,0),field1=mvindex(_raw,1),field2=mvindex(_raw,2),field3=mvindex(_raw,3),field4=mvindex(_raw,4),field5=mvindex(_raw,5),field6=mvindex(_raw,6),field7=mvindex(_raw,7)

Here in <base search> include the query from where you are getting events/raw text like index=indexname

Re: How can I do field extraction on a security log using regex?

493669 — Thu, 15 Feb 2018 15:01:14 GMT

use like this:

index=A |makemv _raw delim=","|eval field0=mvindex(_raw,0),field1=mvindex(_raw,1),field2=mvindex(_raw,2),field3=mvindex(_raw,3),field4=mvindex(_raw,4),field5=mvindex(_raw,5),field6=mvindex(_raw,6),field7=mvindex(_raw,7)