Splunk Search

Extract multiple key value pairs from one of the partially auto extracted field

Path Finder


I am having difficulty in extracting key=value pairs from one of the auto extracted field.
The problem is that, this field may contain just a text value but also could contain multiple key=value pairs in it, so whenever there are multiple key=value pairs in the event then I am not getting the desired results.

Following are some of  my _raw events  -

2021-08-10T11:35:00.505 ip= id=1 event="passed" model="t1" conn="connmsg=\"controller.conn_download::message.clean\", file=\"/home/folder1/filename_8555c5s.ext\", time=\"21:22:02\", day=\"08/24/2021\"" 

2021-08-10T11:35:00.508 ip= id=1 event="running" model="t1" conn="connmsg=\"model.log::option.event.view.log_view_conn, connname=\"model.log::option.event.view.log_view_conn_name\", user=\"xyz\", remote_conn=, auth_conn=\"Base\""

2021-08-10T11:35:00.515 ip= id=1 event="failed" model="t1" conn="Failed to connect to the file for \"file_name\""

2021-08-10T11:35:00.890 ip= id=1 event="extracting" model="t1" conn="connmsg=\"model.log::option.event.view.logout.message\", user=, job_id=65, report_name=",  path=\"{\"type\":1,\"appIds\":\"\",\"path\":\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\/ip_initiate\\/\\/http_code\\/200\",\"restrict\":null}\""

2021-08-10T11:36:00.090 ip= id=1 event="extracting" model="t1" conn="connmsg=\"model.log::option.event.view.audit.message, user=\"qic\\abc_pqr\, reason_msg=\"component.auth::message:unknown_user\", path=/abc/flows/timespan/2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00/ip_initiate/"

2021-08-10T11:36:00.380 ip= id=1 event="triggered" model="t1" conn="Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'"

2021-08-10T11:36:00.880 ip= id=1 event="triggered" model="t1" conn="connmsg=\"model.log::option.event.report.finished\", user=, job_id=65, report_name=",  path=\"{\"type\":1,\"namespace\":\"flows\",\"appIds\":\"10,11,12\",\"path_bar\":\"[\\\"ip_initiate=\\\"]\",\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\/ip_initiate\\/\\/http_code\\/200\",\"restrict\":null}\""

The field which I am facing issue is "conn" field and I want data to be extracted in conn field in somewhat below manner -

Failed to connect to the file for \"file_name\"
"Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'"

but currently its just extracting the next value coming after conn= ,so basically current data in my conn field based on the above raw events looks like -

Failed to connect to the file for \"file_name\"
Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'

The "conn" field might contain even more key value pairs , so also wanted to know if there is some dynamic way to capture if any new key value pair pops in conn field other than those specified ?

Also along with that, the other key value pairs in conn field is sometimes getting auto extracted and sometime it isn't.

I am trying to write Search time field extraction using props and transforms but no luck so far in getting what I want, can someone please help ?

Thanks in Advance.

Labels (3)
0 Karma


You could do something like this (after the blanklines)

| makeresults
| eval _raw="2021-08-10T11:35:00.505 ip= id=1 event=\"passed\" model=\"t1\" conn=\"connmsg=\\\"controller.conn_download::message.clean\\\", file=\\\"/home/folder1/filename_8555c5s.ext\\\", time=\\\"21:22:02\\\", day=\\\"08/24/2021\\\"\" 

2021-08-10T11:35:00.508 ip= id=1 event=\"running\" model=\"t1\" conn=\"connmsg=\\\"model.log::option.event.view.log_view_conn, connname=\\\"model.log::option.event.view.log_view_conn_name\\\", user=\\\"xyz\\\", remote_conn=, auth_conn=\\\"Base\\\"\"

2021-08-10T11:35:00.515 ip= id=1 event=\"failed\" model=\"t1\" conn=\"Failed to connect to the file for \\\"file_name\\\"\"

2021-08-10T11:35:00.890 ip= id=1 event=\"extracting\" model=\"t1\" conn=\"connmsg=\\\"model.log::option.event.view.logout.message\\\", user=, job_id=65, report_name=\",  path=\\\"{\\\"type\\\":1,\\\"appIds\\\":\\\"\\\",\\\"path\\\":\\\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\\\/ip_initiate\\\\/\\\\/http_code\\\\/200\\\",\\\"restrict\\\":null}\\\"\"

2021-08-10T11:36:00.090 ip= id=1 event=\"extracting\" model=\"t1\" conn=\"connmsg=\\\"model.log::option.event.view.audit.message, user=\\\"qic\\\\abc_pqr\\, reason_msg=\\\"component.auth::message:unknown_user\\\", path=/abc/flows/timespan/2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00/ip_initiate/\"

2021-08-10T11:36:00.380 ip= id=1 event=\"triggered\" model=\"t1\" conn=\"Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'\"

2021-08-10T11:36:00.880 ip= id=1 event=\"triggered\" model=\"t1\" conn=\"connmsg=\\\"model.log::option.event.report.finished\\\", user=, job_id=65, report_name=\",  path=\\\"{\\\"type\\\":1,\\\"namespace\\\":\\\"flows\\\",\\\"appIds\\\":\\\"10,11,12\\\",\\\"path_bar\\\":\\\"[\\\\\\\"ip_initiate=\\\\\\\"]\\\",\\\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\\\/ip_initiate\\\\/\\\\/http_code\\\\/200\\\",\\\"restrict\\\":null}\\\"\""
| multikv noheader=t
| table _raw

| rex max_match=0 "(?<_name>\w+)=(?<_value>(\".+?\"|[^\",]+?))(?<!\\\\\")(?=\s+\w+=|\s?$)"
| streamstats count as _event 
| eval index=mvrange(0,mvcount(_name))
| mvexpand index
| eval _name=mvindex(_name,index)
| eval _value=mvindex(_value,index)
| rex field=_value mode=sed "s/^\"//g s/\"$//g"
| eval {_name}=_value
| fields - _name _value index
| stats values(*) as * by _event
| fields - _event
| eval _raw=replace(conn,"\\\\","")
| extract
0 Karma

Path Finder


Thanks, I tried this but its missing out on some of the fields due to _raw data format being inconsistent.

Like for e.g- field report_name its giving me path=  but actually that field is empty.

Also I am looking to extract this using props & transforms.


Tags (1)
0 Karma


What are the props.conf settings for that sourcetype?  What other search-time settings have you tried?

If this reply helps you, an upvote would be appreciated.
0 Karma

Path Finder


Basically I was trying something like -




REGEX = conn="connmsg=([^,]*)|^conn=(.*)
MV_ADD = true
FORMAT = conn::$1::$2

But this doesn't seem to be working as REGEX/FORMAT is not extracting two values for a single field.

So currently I have figured out an alternative way where I am first extracting the entire conn field and then other key value pairs in following way :-

REPORT-extract_conn=extract_conn,extract_connmsg,extract_user ...

REGEX = conn="(.*)"
MV_ADD = true
FORMAT = conn::$1

REGEX = conn="connmsg=([^,]*)
MV_ADD = true
FORMAT = connmsg::$1

REGEX = conn=.*user=([^,]*)
MV_ADD = true
FORMAT = user::$1

.. and so on...

If you can think of or suggest any better and more efficient way to this than i will be happy to try that too.


0 Karma