Splunk Search

Extract multiple key value pairs from one of the partially auto extracted field

ak9092
Path Finder

Hi,

I am having difficulty in extracting key=value pairs from one of the auto extracted field.
The problem is that, this field may contain just a text value but also could contain multiple key=value pairs in it, so whenever there are multiple key=value pairs in the event then I am not getting the desired results.

Following are some of  my _raw events  -

2021-08-10T11:35:00.505 ip=10.1.10.10 id=1 event="passed" model="t1" conn="connmsg=\"controller.conn_download::message.clean\", file=\"/home/folder1/filename_8555c5s.ext\", time=\"21:22:02\", day=\"08/24/2021\"" 

2021-08-10T11:35:00.508 ip=10.1.10.10 id=1 event="running" model="t1" conn="connmsg=\"model.log::option.event.view.log_view_conn, connname=\"model.log::option.event.view.log_view_conn_name\", user=\"xyz\", remote_conn=10.23.55.54, auth_conn=\"Base\""

2021-08-10T11:35:00.515 ip=10.1.10.10 id=1 event="failed" model="t1" conn="Failed to connect to the file for \"file_name\""

2021-08-10T11:35:00.890 ip=10.1.10.10 id=1 event="extracting" model="t1" conn="connmsg=\"model.log::option.event.view.logout.message\", user=, job_id=65, report_name=",  path=\"{\"type\":1,\"appIds\":\"\",\"path\":\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\/ip_initiate\\/10.1.120.11\\/http_code\\/200\",\"restrict\":null}\""

2021-08-10T11:36:00.090 ip=10.1.10.10 id=1 event="extracting" model="t1" conn="connmsg=\"model.log::option.event.view.audit.message, user=\"qic\\abc_pqr\, reason_msg=\"component.auth::message:unknown_user\", path=/abc/flows/timespan/2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00/ip_initiate/10.101.10.20/data.ext"

2021-08-10T11:36:00.380 ip=10.1.10.10 id=1 event="triggered" model="t1" conn="Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'"

2021-08-10T11:36:00.880 ip=10.1.10.10 id=1 event="triggered" model="t1" conn="connmsg=\"model.log::option.event.report.finished\", user=, job_id=65, report_name=",  path=\"{\"type\":1,\"namespace\":\"flows\",\"appIds\":\"10,11,12\",\"path_bar\":\"[\\\"ip_initiate=10.1.120.11\\\"]\",\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\/ip_initiate\\/10.1.120.11\\/http_code\\/200\",\"restrict\":null}\""


The field which I am facing issue is "conn" field and I want data to be extracted in conn field in somewhat below manner -

  conn
\"controller.conn_download::message.clean\"
model.log::option.event.view.log_view_conn
Failed to connect to the file for \"file_name\"
\"model.log::option.event.view.logout.message\"
\"model.log::option.event.view.audit.message\"
"Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'"

but currently its just extracting the next value coming after conn= ,so basically current data in my conn field based on the above raw events looks like -

conn
connmsg=\
connmsg=\
Failed to connect to the file for \"file_name\"
connmsg=\
connmsg=\
Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'

The "conn" field might contain even more key value pairs , so also wanted to know if there is some dynamic way to capture if any new key value pair pops in conn field other than those specified ?

Also along with that, the other key value pairs in conn field is sometimes getting auto extracted and sometime it isn't.

I am trying to write Search time field extraction using props and transforms but no luck so far in getting what I want, can someone please help ?

Thanks in Advance.

Labels (3)
0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

You could do something like this (after the blanklines)

| makeresults
| eval _raw="2021-08-10T11:35:00.505 ip=10.1.10.10 id=1 event=\"passed\" model=\"t1\" conn=\"connmsg=\\\"controller.conn_download::message.clean\\\", file=\\\"/home/folder1/filename_8555c5s.ext\\\", time=\\\"21:22:02\\\", day=\\\"08/24/2021\\\"\" 

2021-08-10T11:35:00.508 ip=10.1.10.10 id=1 event=\"running\" model=\"t1\" conn=\"connmsg=\\\"model.log::option.event.view.log_view_conn, connname=\\\"model.log::option.event.view.log_view_conn_name\\\", user=\\\"xyz\\\", remote_conn=10.23.55.54, auth_conn=\\\"Base\\\"\"

2021-08-10T11:35:00.515 ip=10.1.10.10 id=1 event=\"failed\" model=\"t1\" conn=\"Failed to connect to the file for \\\"file_name\\\"\"

2021-08-10T11:35:00.890 ip=10.1.10.10 id=1 event=\"extracting\" model=\"t1\" conn=\"connmsg=\\\"model.log::option.event.view.logout.message\\\", user=, job_id=65, report_name=\",  path=\\\"{\\\"type\\\":1,\\\"appIds\\\":\\\"\\\",\\\"path\\\":\\\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\\\/ip_initiate\\\\/10.1.120.11\\\\/http_code\\\\/200\\\",\\\"restrict\\\":null}\\\"\"

2021-08-10T11:36:00.090 ip=10.1.10.10 id=1 event=\"extracting\" model=\"t1\" conn=\"connmsg=\\\"model.log::option.event.view.audit.message, user=\\\"qic\\\\abc_pqr\\, reason_msg=\\\"component.auth::message:unknown_user\\\", path=/abc/flows/timespan/2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00/ip_initiate/10.101.10.20/data.ext\"

2021-08-10T11:36:00.380 ip=10.1.10.10 id=1 event=\"triggered\" model=\"t1\" conn=\"Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'\"

2021-08-10T11:36:00.880 ip=10.1.10.10 id=1 event=\"triggered\" model=\"t1\" conn=\"connmsg=\\\"model.log::option.event.report.finished\\\", user=, job_id=65, report_name=\",  path=\\\"{\\\"type\\\":1,\\\"namespace\\\":\\\"flows\\\",\\\"appIds\\\":\\\"10,11,12\\\",\\\"path_bar\\\":\\\"[\\\\\\\"ip_initiate=10.1.120.11\\\\\\\"]\\\",\\\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\\\/ip_initiate\\\\/10.1.120.11\\\\/http_code\\\\/200\\\",\\\"restrict\\\":null}\\\"\""
| multikv noheader=t
| table _raw



| rex max_match=0 "(?<_name>\w+)=(?<_value>(\".+?\"|[^\",]+?))(?<!\\\\\")(?=\s+\w+=|\s?$)"
| streamstats count as _event 
| eval index=mvrange(0,mvcount(_name))
| mvexpand index
| eval _name=mvindex(_name,index)
| eval _value=mvindex(_value,index)
| rex field=_value mode=sed "s/^\"//g s/\"$//g"
| eval {_name}=_value
| fields - _name _value index
| stats values(*) as * by _event
| fields - _event
| eval _raw=replace(conn,"\\\\","")
| extract
0 Karma

ak9092
Path Finder

@ITWhisperer 

Thanks, I tried this but its missing out on some of the fields due to _raw data format being inconsistent.

Like for e.g- field report_name its giving me path=  but actually that field is empty.

Also I am looking to extract this using props & transforms.

Thanks.

Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

What are the props.conf settings for that sourcetype?  What other search-time settings have you tried?

---
If this reply helps you, an upvote would be appreciated.
0 Karma

ak9092
Path Finder

@richgalloway  

Basically I was trying something like -

props.conf

[my_srctype]
REPORT-extract_conn=extract_conn

tranforms.conf

[extract_conn]
REGEX = conn="connmsg=([^,]*)|^conn=(.*)
MV_ADD = true
FORMAT = conn::$1::$2

But this doesn't seem to be working as REGEX/FORMAT is not extracting two values for a single field.

So currently I have figured out an alternative way where I am first extracting the entire conn field and then other key value pairs in following way :-

[my_srctype]
REPORT-extract_conn=extract_conn,extract_connmsg,extract_user ...

[extract_conn]
REGEX = conn="(.*)"
MV_ADD = true
FORMAT = conn::$1

[extract_connmsg]
REGEX = conn="connmsg=([^,]*)
MV_ADD = true
FORMAT = connmsg::$1

[extract_user]
REGEX = conn=.*user=([^,]*)
MV_ADD = true
FORMAT = user::$1

.. and so on...

If you can think of or suggest any better and more efficient way to this than i will be happy to try that too.

Thanks

0 Karma