Splunk Search

Extract multiple key value pairs from one of the partially auto extracted field

ak9092
Path Finder

Hi,

I am having difficulty in extracting key=value pairs from one of the auto extracted field.
The problem is that, this field may contain just a text value but also could contain multiple key=value pairs in it, so whenever there are multiple key=value pairs in the event then I am not getting the desired results.

Following are some of  my _raw events  -

2021-08-10T11:35:00.505 ip=10.1.10.10 id=1 event="passed" model="t1" conn="connmsg=\"controller.conn_download::message.clean\", file=\"/home/folder1/filename_8555c5s.ext\", time=\"21:22:02\", day=\"08/24/2021\"" 

2021-08-10T11:35:00.508 ip=10.1.10.10 id=1 event="running" model="t1" conn="connmsg=\"model.log::option.event.view.log_view_conn, connname=\"model.log::option.event.view.log_view_conn_name\", user=\"xyz\", remote_conn=10.23.55.54, auth_conn=\"Base\""

2021-08-10T11:35:00.515 ip=10.1.10.10 id=1 event="failed" model="t1" conn="Failed to connect to the file for \"file_name\""

2021-08-10T11:35:00.890 ip=10.1.10.10 id=1 event="extracting" model="t1" conn="connmsg=\"model.log::option.event.view.logout.message\", user=, job_id=65, report_name=",  path=\"{\"type\":1,\"appIds\":\"\",\"path\":\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\/ip_initiate\\/10.1.120.11\\/http_code\\/200\",\"restrict\":null}\""

2021-08-10T11:36:00.090 ip=10.1.10.10 id=1 event="extracting" model="t1" conn="connmsg=\"model.log::option.event.view.audit.message, user=\"qic\\abc_pqr\, reason_msg=\"component.auth::message:unknown_user\", path=/abc/flows/timespan/2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00/ip_initiate/10.101.10.20/data.ext"

2021-08-10T11:36:00.380 ip=10.1.10.10 id=1 event="triggered" model="t1" conn="Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'"

2021-08-10T11:36:00.880 ip=10.1.10.10 id=1 event="triggered" model="t1" conn="connmsg=\"model.log::option.event.report.finished\", user=, job_id=65, report_name=",  path=\"{\"type\":1,\"namespace\":\"flows\",\"appIds\":\"10,11,12\",\"path_bar\":\"[\\\"ip_initiate=10.1.120.11\\\"]\",\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\/ip_initiate\\/10.1.120.11\\/http_code\\/200\",\"restrict\":null}\""


The field which I am facing issue is "conn" field and I want data to be extracted in conn field in somewhat below manner -

  conn
\"controller.conn_download::message.clean\"
model.log::option.event.view.log_view_conn
Failed to connect to the file for \"file_name\"
\"model.log::option.event.view.logout.message\"
\"model.log::option.event.view.audit.message\"
"Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'"

but currently its just extracting the next value coming after conn= ,so basically current data in my conn field based on the above raw events looks like -

conn
connmsg=\
connmsg=\
Failed to connect to the file for \"file_name\"
connmsg=\
connmsg=\
Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'

The "conn" field might contain even more key value pairs , so also wanted to know if there is some dynamic way to capture if any new key value pair pops in conn field other than those specified ?

Also along with that, the other key value pairs in conn field is sometimes getting auto extracted and sometime it isn't.

I am trying to write Search time field extraction using props and transforms but no luck so far in getting what I want, can someone please help ?

Thanks in Advance.

Labels (3)
0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

You could do something like this (after the blanklines)

| makeresults
| eval _raw="2021-08-10T11:35:00.505 ip=10.1.10.10 id=1 event=\"passed\" model=\"t1\" conn=\"connmsg=\\\"controller.conn_download::message.clean\\\", file=\\\"/home/folder1/filename_8555c5s.ext\\\", time=\\\"21:22:02\\\", day=\\\"08/24/2021\\\"\" 

2021-08-10T11:35:00.508 ip=10.1.10.10 id=1 event=\"running\" model=\"t1\" conn=\"connmsg=\\\"model.log::option.event.view.log_view_conn, connname=\\\"model.log::option.event.view.log_view_conn_name\\\", user=\\\"xyz\\\", remote_conn=10.23.55.54, auth_conn=\\\"Base\\\"\"

2021-08-10T11:35:00.515 ip=10.1.10.10 id=1 event=\"failed\" model=\"t1\" conn=\"Failed to connect to the file for \\\"file_name\\\"\"

2021-08-10T11:35:00.890 ip=10.1.10.10 id=1 event=\"extracting\" model=\"t1\" conn=\"connmsg=\\\"model.log::option.event.view.logout.message\\\", user=, job_id=65, report_name=\",  path=\\\"{\\\"type\\\":1,\\\"appIds\\\":\\\"\\\",\\\"path\\\":\\\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\\\/ip_initiate\\\\/10.1.120.11\\\\/http_code\\\\/200\\\",\\\"restrict\\\":null}\\\"\"

2021-08-10T11:36:00.090 ip=10.1.10.10 id=1 event=\"extracting\" model=\"t1\" conn=\"connmsg=\\\"model.log::option.event.view.audit.message, user=\\\"qic\\\\abc_pqr\\, reason_msg=\\\"component.auth::message:unknown_user\\\", path=/abc/flows/timespan/2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00/ip_initiate/10.101.10.20/data.ext\"

2021-08-10T11:36:00.380 ip=10.1.10.10 id=1 event=\"triggered\" model=\"t1\" conn=\"Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'\"

2021-08-10T11:36:00.880 ip=10.1.10.10 id=1 event=\"triggered\" model=\"t1\" conn=\"connmsg=\\\"model.log::option.event.report.finished\\\", user=, job_id=65, report_name=\",  path=\\\"{\\\"type\\\":1,\\\"namespace\\\":\\\"flows\\\",\\\"appIds\\\":\\\"10,11,12\\\",\\\"path_bar\\\":\\\"[\\\\\\\"ip_initiate=10.1.120.11\\\\\\\"]\\\",\\\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\\\/ip_initiate\\\\/10.1.120.11\\\\/http_code\\\\/200\\\",\\\"restrict\\\":null}\\\"\""
| multikv noheader=t
| table _raw



| rex max_match=0 "(?<_name>\w+)=(?<_value>(\".+?\"|[^\",]+?))(?<!\\\\\")(?=\s+\w+=|\s?$)"
| streamstats count as _event 
| eval index=mvrange(0,mvcount(_name))
| mvexpand index
| eval _name=mvindex(_name,index)
| eval _value=mvindex(_value,index)
| rex field=_value mode=sed "s/^\"//g s/\"$//g"
| eval {_name}=_value
| fields - _name _value index
| stats values(*) as * by _event
| fields - _event
| eval _raw=replace(conn,"\\\\","")
| extract
0 Karma

ak9092
Path Finder

@ITWhisperer 

Thanks, I tried this but its missing out on some of the fields due to _raw data format being inconsistent.

Like for e.g- field report_name its giving me path=  but actually that field is empty.

Also I am looking to extract this using props & transforms.

Thanks.

Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

What are the props.conf settings for that sourcetype?  What other search-time settings have you tried?

---
If this reply helps you, Karma would be appreciated.
0 Karma

ak9092
Path Finder

@richgalloway  

Basically I was trying something like -

props.conf

[my_srctype]
REPORT-extract_conn=extract_conn

tranforms.conf

[extract_conn]
REGEX = conn="connmsg=([^,]*)|^conn=(.*)
MV_ADD = true
FORMAT = conn::$1::$2

But this doesn't seem to be working as REGEX/FORMAT is not extracting two values for a single field.

So currently I have figured out an alternative way where I am first extracting the entire conn field and then other key value pairs in following way :-

[my_srctype]
REPORT-extract_conn=extract_conn,extract_connmsg,extract_user ...

[extract_conn]
REGEX = conn="(.*)"
MV_ADD = true
FORMAT = conn::$1

[extract_connmsg]
REGEX = conn="connmsg=([^,]*)
MV_ADD = true
FORMAT = connmsg::$1

[extract_user]
REGEX = conn=.*user=([^,]*)
MV_ADD = true
FORMAT = user::$1

.. and so on...

If you can think of or suggest any better and more efficient way to this than i will be happy to try that too.

Thanks

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...