Splunk Search

Extract multiple key value pairs from one of the partially auto extracted field

Path Finder


I am having difficulty in extracting key=value pairs from one of the auto extracted field.
The problem is that, this field may contain just a text value but also could contain multiple key=value pairs in it, so whenever there are multiple key=value pairs in the event then I am not getting the desired results.

Following are some of  my _raw events  -

2021-08-10T11:35:00.505 ip= id=1 event="passed" model="t1" conn="connmsg=\"controller.conn_download::message.clean\", file=\"/home/folder1/filename_8555c5s.ext\", time=\"21:22:02\", day=\"08/24/2021\"" 

2021-08-10T11:35:00.508 ip= id=1 event="running" model="t1" conn="connmsg=\"model.log::option.event.view.log_view_conn, connname=\"model.log::option.event.view.log_view_conn_name\", user=\"xyz\", remote_conn=, auth_conn=\"Base\""

2021-08-10T11:35:00.515 ip= id=1 event="failed" model="t1" conn="Failed to connect to the file for \"file_name\""

2021-08-10T11:35:00.890 ip= id=1 event="extracting" model="t1" conn="connmsg=\"model.log::option.event.view.logout.message\", user=, job_id=65, report_name=",  path=\"{\"type\":1,\"appIds\":\"\",\"path\":\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\/ip_initiate\\/\\/http_code\\/200\",\"restrict\":null}\""

2021-08-10T11:36:00.090 ip= id=1 event="extracting" model="t1" conn="connmsg=\"model.log::option.event.view.audit.message, user=\"qic\\abc_pqr\, reason_msg=\"component.auth::message:unknown_user\", path=/abc/flows/timespan/2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00/ip_initiate/"

2021-08-10T11:36:00.380 ip= id=1 event="triggered" model="t1" conn="Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'"

2021-08-10T11:36:00.880 ip= id=1 event="triggered" model="t1" conn="connmsg=\"model.log::option.event.report.finished\", user=, job_id=65, report_name=",  path=\"{\"type\":1,\"namespace\":\"flows\",\"appIds\":\"10,11,12\",\"path_bar\":\"[\\\"ip_initiate=\\\"]\",\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\/ip_initiate\\/\\/http_code\\/200\",\"restrict\":null}\""

The field which I am facing issue is "conn" field and I want data to be extracted in conn field in somewhat below manner -

Failed to connect to the file for \"file_name\"
"Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'"

but currently its just extracting the next value coming after conn= ,so basically current data in my conn field based on the above raw events looks like -

Failed to connect to the file for \"file_name\"
Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'

The "conn" field might contain even more key value pairs , so also wanted to know if there is some dynamic way to capture if any new key value pair pops in conn field other than those specified ?

Also along with that, the other key value pairs in conn field is sometimes getting auto extracted and sometime it isn't.

I am trying to write Search time field extraction using props and transforms but no luck so far in getting what I want, can someone please help ?

Thanks in Advance.

Labels (3)
0 Karma


You could do something like this (after the blanklines)

| makeresults
| eval _raw="2021-08-10T11:35:00.505 ip= id=1 event=\"passed\" model=\"t1\" conn=\"connmsg=\\\"controller.conn_download::message.clean\\\", file=\\\"/home/folder1/filename_8555c5s.ext\\\", time=\\\"21:22:02\\\", day=\\\"08/24/2021\\\"\" 

2021-08-10T11:35:00.508 ip= id=1 event=\"running\" model=\"t1\" conn=\"connmsg=\\\"model.log::option.event.view.log_view_conn, connname=\\\"model.log::option.event.view.log_view_conn_name\\\", user=\\\"xyz\\\", remote_conn=, auth_conn=\\\"Base\\\"\"

2021-08-10T11:35:00.515 ip= id=1 event=\"failed\" model=\"t1\" conn=\"Failed to connect to the file for \\\"file_name\\\"\"

2021-08-10T11:35:00.890 ip= id=1 event=\"extracting\" model=\"t1\" conn=\"connmsg=\\\"model.log::option.event.view.logout.message\\\", user=, job_id=65, report_name=\",  path=\\\"{\\\"type\\\":1,\\\"appIds\\\":\\\"\\\",\\\"path\\\":\\\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\\\/ip_initiate\\\\/\\\\/http_code\\\\/200\\\",\\\"restrict\\\":null}\\\"\"

2021-08-10T11:36:00.090 ip= id=1 event=\"extracting\" model=\"t1\" conn=\"connmsg=\\\"model.log::option.event.view.audit.message, user=\\\"qic\\\\abc_pqr\\, reason_msg=\\\"component.auth::message:unknown_user\\\", path=/abc/flows/timespan/2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00/ip_initiate/\"

2021-08-10T11:36:00.380 ip= id=1 event=\"triggered\" model=\"t1\" conn=\"Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'\"

2021-08-10T11:36:00.880 ip= id=1 event=\"triggered\" model=\"t1\" conn=\"connmsg=\\\"model.log::option.event.report.finished\\\", user=, job_id=65, report_name=\",  path=\\\"{\\\"type\\\":1,\\\"namespace\\\":\\\"flows\\\",\\\"appIds\\\":\\\"10,11,12\\\",\\\"path_bar\\\":\\\"[\\\\\\\"ip_initiate=\\\\\\\"]\\\",\\\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\\\/ip_initiate\\\\/\\\\/http_code\\\\/200\\\",\\\"restrict\\\":null}\\\"\""
| multikv noheader=t
| table _raw

| rex max_match=0 "(?<_name>\w+)=(?<_value>(\".+?\"|[^\",]+?))(?<!\\\\\")(?=\s+\w+=|\s?$)"
| streamstats count as _event 
| eval index=mvrange(0,mvcount(_name))
| mvexpand index
| eval _name=mvindex(_name,index)
| eval _value=mvindex(_value,index)
| rex field=_value mode=sed "s/^\"//g s/\"$//g"
| eval {_name}=_value
| fields - _name _value index
| stats values(*) as * by _event
| fields - _event
| eval _raw=replace(conn,"\\\\","")
| extract
0 Karma

Path Finder


Thanks, I tried this but its missing out on some of the fields due to _raw data format being inconsistent.

Like for e.g- field report_name its giving me path=  but actually that field is empty.

Also I am looking to extract this using props & transforms.


Tags (1)
0 Karma


What are the props.conf settings for that sourcetype?  What other search-time settings have you tried?

If this reply helps you, Karma would be appreciated.
0 Karma

Path Finder


Basically I was trying something like -




REGEX = conn="connmsg=([^,]*)|^conn=(.*)
MV_ADD = true
FORMAT = conn::$1::$2

But this doesn't seem to be working as REGEX/FORMAT is not extracting two values for a single field.

So currently I have figured out an alternative way where I am first extracting the entire conn field and then other key value pairs in following way :-

REPORT-extract_conn=extract_conn,extract_connmsg,extract_user ...

REGEX = conn="(.*)"
MV_ADD = true
FORMAT = conn::$1

REGEX = conn="connmsg=([^,]*)
MV_ADD = true
FORMAT = connmsg::$1

REGEX = conn=.*user=([^,]*)
MV_ADD = true
FORMAT = user::$1

.. and so on...

If you can think of or suggest any better and more efficient way to this than i will be happy to try that too.


0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...