Hi,
I am having difficulty in extracting key=value pairs from one of the auto extracted field.
The problem is that, this field may contain just a text value but also could contain multiple key=value pairs in it, so whenever there are multiple key=value pairs in the event then I am not getting the desired results.
Following are some of my _raw events -
2021-08-10T11:35:00.505 ip=10.1.10.10 id=1 event="passed" model="t1" conn="connmsg=\"controller.conn_download::message.clean\", file=\"/home/folder1/filename_8555c5s.ext\", time=\"21:22:02\", day=\"08/24/2021\""
2021-08-10T11:35:00.508 ip=10.1.10.10 id=1 event="running" model="t1" conn="connmsg=\"model.log::option.event.view.log_view_conn, connname=\"model.log::option.event.view.log_view_conn_name\", user=\"xyz\", remote_conn=10.23.55.54, auth_conn=\"Base\""
2021-08-10T11:35:00.515 ip=10.1.10.10 id=1 event="failed" model="t1" conn="Failed to connect to the file for \"file_name\""
2021-08-10T11:35:00.890 ip=10.1.10.10 id=1 event="extracting" model="t1" conn="connmsg=\"model.log::option.event.view.logout.message\", user=, job_id=65, report_name=", path=\"{\"type\":1,\"appIds\":\"\",\"path\":\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\/ip_initiate\\/10.1.120.11\\/http_code\\/200\",\"restrict\":null}\""
2021-08-10T11:36:00.090 ip=10.1.10.10 id=1 event="extracting" model="t1" conn="connmsg=\"model.log::option.event.view.audit.message, user=\"qic\\abc_pqr\, reason_msg=\"component.auth::message:unknown_user\", path=/abc/flows/timespan/2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00/ip_initiate/10.101.10.20/data.ext"
2021-08-10T11:36:00.380 ip=10.1.10.10 id=1 event="triggered" model="t1" conn="Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'"
2021-08-10T11:36:00.880 ip=10.1.10.10 id=1 event="triggered" model="t1" conn="connmsg=\"model.log::option.event.report.finished\", user=, job_id=65, report_name=", path=\"{\"type\":1,\"namespace\":\"flows\",\"appIds\":\"10,11,12\",\"path_bar\":\"[\\\"ip_initiate=10.1.120.11\\\"]\",\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\/ip_initiate\\/10.1.120.11\\/http_code\\/200\",\"restrict\":null}\""
The field which I am facing issue is "conn" field and I want data to be extracted in conn field in somewhat below manner -
conn
\"controller.conn_download::message.clean\"
model.log::option.event.view.log_view_conn
Failed to connect to the file for \"file_name\"
\"model.log::option.event.view.logout.message\"
\"model.log::option.event.view.audit.message\"
"Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'"
but currently its just extracting the next value coming after conn= ,so basically current data in my conn field based on the above raw events looks like -
conn
connmsg=\
connmsg=\
Failed to connect to the file for \"file_name\"
connmsg=\
connmsg=\
Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'
The "conn" field might contain even more key value pairs , so also wanted to know if there is some dynamic way to capture if any new key value pair pops in conn field other than those specified ?
Also along with that, the other key value pairs in conn field is sometimes getting auto extracted and sometime it isn't.
I am trying to write Search time field extraction using props and transforms but no luck so far in getting what I want, can someone please help ?
Thanks in Advance.
You could do something like this (after the blanklines)
| makeresults
| eval _raw="2021-08-10T11:35:00.505 ip=10.1.10.10 id=1 event=\"passed\" model=\"t1\" conn=\"connmsg=\\\"controller.conn_download::message.clean\\\", file=\\\"/home/folder1/filename_8555c5s.ext\\\", time=\\\"21:22:02\\\", day=\\\"08/24/2021\\\"\"
2021-08-10T11:35:00.508 ip=10.1.10.10 id=1 event=\"running\" model=\"t1\" conn=\"connmsg=\\\"model.log::option.event.view.log_view_conn, connname=\\\"model.log::option.event.view.log_view_conn_name\\\", user=\\\"xyz\\\", remote_conn=10.23.55.54, auth_conn=\\\"Base\\\"\"
2021-08-10T11:35:00.515 ip=10.1.10.10 id=1 event=\"failed\" model=\"t1\" conn=\"Failed to connect to the file for \\\"file_name\\\"\"
2021-08-10T11:35:00.890 ip=10.1.10.10 id=1 event=\"extracting\" model=\"t1\" conn=\"connmsg=\\\"model.log::option.event.view.logout.message\\\", user=, job_id=65, report_name=\", path=\\\"{\\\"type\\\":1,\\\"appIds\\\":\\\"\\\",\\\"path\\\":\\\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\\\/ip_initiate\\\\/10.1.120.11\\\\/http_code\\\\/200\\\",\\\"restrict\\\":null}\\\"\"
2021-08-10T11:36:00.090 ip=10.1.10.10 id=1 event=\"extracting\" model=\"t1\" conn=\"connmsg=\\\"model.log::option.event.view.audit.message, user=\\\"qic\\\\abc_pqr\\, reason_msg=\\\"component.auth::message:unknown_user\\\", path=/abc/flows/timespan/2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00/ip_initiate/10.101.10.20/data.ext\"
2021-08-10T11:36:00.380 ip=10.1.10.10 id=1 event=\"triggered\" model=\"t1\" conn=\"Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'\"
2021-08-10T11:36:00.880 ip=10.1.10.10 id=1 event=\"triggered\" model=\"t1\" conn=\"connmsg=\\\"model.log::option.event.report.finished\\\", user=, job_id=65, report_name=\", path=\\\"{\\\"type\\\":1,\\\"namespace\\\":\\\"flows\\\",\\\"appIds\\\":\\\"10,11,12\\\",\\\"path_bar\\\":\\\"[\\\\\\\"ip_initiate=10.1.120.11\\\\\\\"]\\\",\\\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\\\/ip_initiate\\\\/10.1.120.11\\\\/http_code\\\\/200\\\",\\\"restrict\\\":null}\\\"\""
| multikv noheader=t
| table _raw
| rex max_match=0 "(?<_name>\w+)=(?<_value>(\".+?\"|[^\",]+?))(?<!\\\\\")(?=\s+\w+=|\s?$)"
| streamstats count as _event
| eval index=mvrange(0,mvcount(_name))
| mvexpand index
| eval _name=mvindex(_name,index)
| eval _value=mvindex(_value,index)
| rex field=_value mode=sed "s/^\"//g s/\"$//g"
| eval {_name}=_value
| fields - _name _value index
| stats values(*) as * by _event
| fields - _event
| eval _raw=replace(conn,"\\\\","")
| extract
Thanks, I tried this but its missing out on some of the fields due to _raw data format being inconsistent.
Like for e.g- field report_name its giving me path= but actually that field is empty.
Also I am looking to extract this using props & transforms.
Thanks.
What are the props.conf settings for that sourcetype? What other search-time settings have you tried?
Basically I was trying something like -
props.conf
[my_srctype]
REPORT-extract_conn=extract_conn
tranforms.conf
[extract_conn]
REGEX = conn="connmsg=([^,]*)|^conn=(.*)
MV_ADD = true
FORMAT = conn::$1::$2
But this doesn't seem to be working as REGEX/FORMAT is not extracting two values for a single field.
So currently I have figured out an alternative way where I am first extracting the entire conn field and then other key value pairs in following way :-
[my_srctype]
REPORT-extract_conn=extract_conn,extract_connmsg,extract_user ...
[extract_conn]
REGEX = conn="(.*)"
MV_ADD = true
FORMAT = conn::$1
[extract_connmsg]
REGEX = conn="connmsg=([^,]*)
MV_ADD = true
FORMAT = connmsg::$1
[extract_user]
REGEX = conn=.*user=([^,]*)
MV_ADD = true
FORMAT = user::$1
.. and so on...
If you can think of or suggest any better and more efficient way to this than i will be happy to try that too.
Thanks