topic Re: Extract key-value pairs in Splunk Search

Extract key-value pairs

scottkoontz57 — Thu, 09 May 2019 23:31:56 GMT

I'm trying to extract the key-value pairs from an Untangle firewall log ( syslog ), but the Regex example I found on the forum don't work and I'm not even sure that's the best approach.
Example:

May  9 16:19:33 192.168.10.254 May  9 16:19:34 INFO  uvm[0]:  {"timeStamp":"2019-05-09 16:19:34.148","s2pBytes":0,"p2sBytes":0,"endTime":1557443974148,"sessionId":102006119003906,"tag":"uvm[0]: ","class":"class com.untangle.uvm.app.SessionStatsEvent","sessionEvent":{"entitled":true,"partitionTablePostfix":"_2019_05_09","hostname":"192.168.10.102","CServerPort":161,"protocol":17,"protocolName":"UDP","tag":"uvm[0]: ","localAddr":"/192.168.10.102","class":"class com.untangle.uvm.app.SessionEvent","SServerAddr":"/192.168.0.192","remoteAddr":"/192.168.0.192","serverIntf":100,"CClientAddr":"/192.168.10.102","serverCountry":"XU","sessionId":102006119003906,"SClientAddr":"/97.115.239.26","clientCountry":"XL","CClientPort":65181,"policyRuleId":0,"timeStamp":"2019-05-09 16:19:34.148","clientIntf":2,"policyId":1,"SClientPort":65181,"bypassed":false,"SServerPort":161,"CServerAddr":"/192.168.0.192","tagsString":""},"c2pBytes":0,"p2cBytes":0,"partitionTablePostfix":"_2019_05_09"}

Re: Extract key-value pairs

richgalloway — Fri, 10 May 2019 03:22:20 GMT

What is the regex you're using?

Re: Extract key-value pairs

scottkoontz57 — Fri, 10 May 2019 04:09:43 GMT

"([^"]+)":"([^"]+)

Re: Extract key-value pairs

sumanssah — Fri, 10 May 2019 04:31:11 GMT

If you are using "syslog" mechanism to send logs from untangle to Splunk, I would suggest using below mentioned SPL for field extraction

index="untangle" 
| rex "(?<json>\{.+)" | spath input=json | fields - json 
| replace /* with * in CClientAddr,CServerAddr,SClientAddr,SServerAddr,localAddr,remoteAddr

Re: Extract key-value pairs

scottkoontz57 — Fri, 10 May 2019 14:15:02 GMT

Can you elaborate? What exactly would I do with the code you provided?

Re: Extract key-value pairs

sumanssah — Thu, 16 May 2019 13:08:00 GMT

this will perform search time field extraction for untangle logs.

Re: Extract key-value pairs

woodcock — Wed, 03 Jul 2019 00:44:22 GMT

Try this:

REGEX = "([^"]*)":(?:(?:{[^{}]+)}|(?:"?[^,"]*)"?)(?=$|,)
FORMAT = $1::$2
MV_ADD = true

See it work here:
https://regex101.com/r/9Iyyy9/1

Re: Extract key-value pairs

venkasplunk — Fri, 12 Jul 2019 04:15:58 GMT

Hello, use this. simple and powerful.

extract pairdelim="{|}" kvdelim="=" | table cb,et,ip,action -----> Here pairdelim is | and kvdelim (keyvalue delim is 😃

For your case, seems like pairdelim is , and kvdelim is :

Re: Extract key-value pairs

landen99 — Sat, 13 Jul 2019 23:33:09 GMT

intransforms.conf

[my_transform]
REGEX = "([^",{}]+)":"?([^",{}]+)"?
FORMAT = $1::$2
MV_ADD = true

add the props to call it: REPORT-mytransform=mytransform.