Splunk Search

Extract multiple key value pairs from one of the partially auto extracted field

ak9092
Path Finder

Hi,

I am having difficulty in extracting key=value pairs from one of the auto extracted field.
The problem is that, this field may contain just a text value but also could contain multiple key=value pairs in it, so whenever there are multiple key=value pairs in the event then I am not getting the desired results.

Following are some of  my _raw events  -

2021-08-10T11:35:00.505 ip=10.1.10.10 id=1 event="passed" model="t1" conn="connmsg=\"controller.conn_download::message.clean\", file=\"/home/folder1/filename_8555c5s.ext\", time=\"21:22:02\", day=\"08/24/2021\"" 

2021-08-10T11:35:00.508 ip=10.1.10.10 id=1 event="running" model="t1" conn="connmsg=\"model.log::option.event.view.log_view_conn, connname=\"model.log::option.event.view.log_view_conn_name\", user=\"xyz\", remote_conn=10.23.55.54, auth_conn=\"Base\""

2021-08-10T11:35:00.515 ip=10.1.10.10 id=1 event="failed" model="t1" conn="Failed to connect to the file for \"file_name\""

2021-08-10T11:35:00.890 ip=10.1.10.10 id=1 event="extracting" model="t1" conn="connmsg=\"model.log::option.event.view.logout.message\", user=, job_id=65, report_name=",  path=\"{\"type\":1,\"appIds\":\"\",\"path\":\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\/ip_initiate\\/10.1.120.11\\/http_code\\/200\",\"restrict\":null}\""

2021-08-10T11:36:00.090 ip=10.1.10.10 id=1 event="extracting" model="t1" conn="connmsg=\"model.log::option.event.view.audit.message, user=\"qic\\abc_pqr\, reason_msg=\"component.auth::message:unknown_user\", path=/abc/flows/timespan/2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00/ip_initiate/10.101.10.20/data.ext"

2021-08-10T11:36:00.380 ip=10.1.10.10 id=1 event="triggered" model="t1" conn="Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'"

2021-08-10T11:36:00.880 ip=10.1.10.10 id=1 event="triggered" model="t1" conn="connmsg=\"model.log::option.event.report.finished\", user=, job_id=65, report_name=",  path=\"{\"type\":1,\"namespace\":\"flows\",\"appIds\":\"10,11,12\",\"path_bar\":\"[\\\"ip_initiate=10.1.120.11\\\"]\",\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\/ip_initiate\\/10.1.120.11\\/http_code\\/200\",\"restrict\":null}\""


The field which I am facing issue is "conn" field and I want data to be extracted in conn field in somewhat below manner -

  conn
\"controller.conn_download::message.clean\"
model.log::option.event.view.log_view_conn
Failed to connect to the file for \"file_name\"
\"model.log::option.event.view.logout.message\"
\"model.log::option.event.view.audit.message\"
"Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'"

but currently its just extracting the next value coming after conn= ,so basically current data in my conn field based on the above raw events looks like -

conn
connmsg=\
connmsg=\
Failed to connect to the file for \"file_name\"
connmsg=\
connmsg=\
Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'

The "conn" field might contain even more key value pairs , so also wanted to know if there is some dynamic way to capture if any new key value pair pops in conn field other than those specified ?

Also along with that, the other key value pairs in conn field is sometimes getting auto extracted and sometime it isn't.

I am trying to write Search time field extraction using props and transforms but no luck so far in getting what I want, can someone please help ?

Thanks in Advance.

Labels (3)
0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

You could do something like this (after the blanklines)

| makeresults
| eval _raw="2021-08-10T11:35:00.505 ip=10.1.10.10 id=1 event=\"passed\" model=\"t1\" conn=\"connmsg=\\\"controller.conn_download::message.clean\\\", file=\\\"/home/folder1/filename_8555c5s.ext\\\", time=\\\"21:22:02\\\", day=\\\"08/24/2021\\\"\" 

2021-08-10T11:35:00.508 ip=10.1.10.10 id=1 event=\"running\" model=\"t1\" conn=\"connmsg=\\\"model.log::option.event.view.log_view_conn, connname=\\\"model.log::option.event.view.log_view_conn_name\\\", user=\\\"xyz\\\", remote_conn=10.23.55.54, auth_conn=\\\"Base\\\"\"

2021-08-10T11:35:00.515 ip=10.1.10.10 id=1 event=\"failed\" model=\"t1\" conn=\"Failed to connect to the file for \\\"file_name\\\"\"

2021-08-10T11:35:00.890 ip=10.1.10.10 id=1 event=\"extracting\" model=\"t1\" conn=\"connmsg=\\\"model.log::option.event.view.logout.message\\\", user=, job_id=65, report_name=\",  path=\\\"{\\\"type\\\":1,\\\"appIds\\\":\\\"\\\",\\\"path\\\":\\\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\\\/ip_initiate\\\\/10.1.120.11\\\\/http_code\\\\/200\\\",\\\"restrict\\\":null}\\\"\"

2021-08-10T11:36:00.090 ip=10.1.10.10 id=1 event=\"extracting\" model=\"t1\" conn=\"connmsg=\\\"model.log::option.event.view.audit.message, user=\\\"qic\\\\abc_pqr\\, reason_msg=\\\"component.auth::message:unknown_user\\\", path=/abc/flows/timespan/2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00/ip_initiate/10.101.10.20/data.ext\"

2021-08-10T11:36:00.380 ip=10.1.10.10 id=1 event=\"triggered\" model=\"t1\" conn=\"Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'\"

2021-08-10T11:36:00.880 ip=10.1.10.10 id=1 event=\"triggered\" model=\"t1\" conn=\"connmsg=\\\"model.log::option.event.report.finished\\\", user=, job_id=65, report_name=\",  path=\\\"{\\\"type\\\":1,\\\"namespace\\\":\\\"flows\\\",\\\"appIds\\\":\\\"10,11,12\\\",\\\"path_bar\\\":\\\"[\\\\\\\"ip_initiate=10.1.120.11\\\\\\\"]\\\",\\\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\\\/ip_initiate\\\\/10.1.120.11\\\\/http_code\\\\/200\\\",\\\"restrict\\\":null}\\\"\""
| multikv noheader=t
| table _raw



| rex max_match=0 "(?<_name>\w+)=(?<_value>(\".+?\"|[^\",]+?))(?<!\\\\\")(?=\s+\w+=|\s?$)"
| streamstats count as _event 
| eval index=mvrange(0,mvcount(_name))
| mvexpand index
| eval _name=mvindex(_name,index)
| eval _value=mvindex(_value,index)
| rex field=_value mode=sed "s/^\"//g s/\"$//g"
| eval {_name}=_value
| fields - _name _value index
| stats values(*) as * by _event
| fields - _event
| eval _raw=replace(conn,"\\\\","")
| extract
0 Karma

ak9092
Path Finder

@ITWhisperer 

Thanks, I tried this but its missing out on some of the fields due to _raw data format being inconsistent.

Like for e.g- field report_name its giving me path=  but actually that field is empty.

Also I am looking to extract this using props & transforms.

Thanks.

Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

What are the props.conf settings for that sourcetype?  What other search-time settings have you tried?

---
If this reply helps you, Karma would be appreciated.
0 Karma

ak9092
Path Finder

@richgalloway  

Basically I was trying something like -

props.conf

[my_srctype]
REPORT-extract_conn=extract_conn

tranforms.conf

[extract_conn]
REGEX = conn="connmsg=([^,]*)|^conn=(.*)
MV_ADD = true
FORMAT = conn::$1::$2

But this doesn't seem to be working as REGEX/FORMAT is not extracting two values for a single field.

So currently I have figured out an alternative way where I am first extracting the entire conn field and then other key value pairs in following way :-

[my_srctype]
REPORT-extract_conn=extract_conn,extract_connmsg,extract_user ...

[extract_conn]
REGEX = conn="(.*)"
MV_ADD = true
FORMAT = conn::$1

[extract_connmsg]
REGEX = conn="connmsg=([^,]*)
MV_ADD = true
FORMAT = connmsg::$1

[extract_user]
REGEX = conn=.*user=([^,]*)
MV_ADD = true
FORMAT = user::$1

.. and so on...

If you can think of or suggest any better and more efficient way to this than i will be happy to try that too.

Thanks

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...