Regex Help

[newbie2tech]

Hi Team,

Need your help with Regex to extract key value pairs. Below is sample event

2017-06-27 14:35:38.000 INFO   [http-nio-0.0.0.0-19752-exec-4 - RequestResponseLoggerFilter] SessionID 2017-06-28-10:31:01-0855306 correlationID:bbacfa29-095d-49c0-b146-fe456h417ecbb681 "Request" : {"top":{"ref":"https://abc.com","adrum":"isAjax:true","content-length":"199","xyz-id":"123456","origin":"https://offline.xyz.com","xyz-authenticationprovider":"kbc","xyz-kbc-entry":"ipad","xyz-originaluri":"/kpi/abcf","via":"http/1.1 xyz.com","xyz-hyt-entrydomain":"mb","xyz-ipaddress":"117.00.109.103","xyz-originatorauthentication":"VUJTLdGhlbnRpY2F0aW9uTW5hbFVyaT0lMkZhcGklMkZ3bWElMkZjcmVkaXQtY2FyZC1wYXltZW50cyUyRnYxJTJGc2NoZWR1bGVkJTJGYWRkJlVCUy1SZXF1ZXN0QmFzZVVybD1odHRwcyU","x-forwarded-host":"offline.xyz.com","xyz-isuseraccessinganyaddressinternally":"false","x-forwarded-prefix":"/api","xyz-isuseraccessinginternally":"false","host":"offline.xyz.com","content-type":"application/json","xyz-hyt-hashregistrationid":"61fa7e9eb9afca3c09412a10fb02986d","accept-language":"en-us","xyz-authenticationlevel":"25","x-forwarded-proto":"https","dnt":"1","x-forwarded-for":"173.66.169.203","xyz-requestbaseurl":"https://offline.xyz.com/api","accept":"*/*","xyz-hyt-registrationid":"123456","x-forwarded-server":"148.112.147.17","singularityheader":"appId=7*ctrlguid=123456*acctguid=3f5d5ee2-41b3-4e59-ad10-de0b93044*ts=1498660534590*btid=2206*id=9497e8be-3275-417b-9893-30dbcca2d0*exitid=2*unresolvedexitid=164*cidfrom=148*etypeorder=CUSTOM*esubtype=CUSTOM*cidto=545","xyz-hyt-sessionid":"2017-06-28-10:31:01-085","xyz-isproxyuser":"false","xyz-isemployee":"false","xyz-hyt-entrypoint":"Mens","accept-encoding":"gzip,deflate","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/603.2.5 (KHTML, like Gecko) Version/10.1.1 Safari/603.2.5","xyz-iscallersecure":"true"},"correlationId":"bbddfa29-09fd-45c0-b136-fe417ecbb681"}

Ideally i would like to break down the above event into ky-value pairs such that i can build stats using the extracted fields, if it makes it any easier, extracting the key:value paris inside the {"ref"..."xyz-iscallersecure":"true"} portion would suffice.

one of the sample stats which i would be needing would look like below..

mysearch | stats count(xyz-id) by xyz-kbc-entry

Can you please help share regex for getting this done. I tried multiple options using my limited regex knowledge but couldn't get it exactly.

Thank you in advance.

[woodcock]

Like this:

| makeresults 
| eval _raw="2017-06-27 14:35:38.000 INFO   [http-nio-0.0.0.0-19752-exec-4 - RequestResponseLoggerFilter] SessionID 2017-06-28-10:31:01-0855306 correlationID:bbacfa29-095d-49c0-b146-fe456h417ecbb681 \"Request\" : {\"top\":{\"ref\":\"https://abc.com\",\"adrum\":\"isAjax:true\",\"content-length\":\"199\",\"xyz-id\":\"123456\",\"origin\":\"https://offline.xyz.com\",\"xyz-authenticationprovider\":\"kbc\",\"xyz-kbc-entry\":\"ipad\",\"xyz-originaluri\":\"/kpi/abcf\",\"via\":\"http/1.1 xyz.com\",\"xyz-hyt-entrydomain\":\"mb\",\"xyz-ipaddress\":\"117.00.109.103\",\"xyz-originatorauthentication\":\"VUJTLdGhlbnRpY2F0aW9uTW5hbFVyaT0lMkZhcGklMkZ3bWElMkZjcmVkaXQtY2FyZC1wYXltZW50cyUyRnYxJTJGc2NoZWR1bGVkJTJGYWRkJlVCUy1SZXF1ZXN0QmFzZVVybD1odHRwcyU\",\"x-forwarded-host\":\"offline.xyz.com\",\"xyz-isuseraccessinganyaddressinternally\":\"false\",\"x-forwarded-prefix\":\"/api\",\"xyz-isuseraccessinginternally\":\"false\",\"host\":\"offline.xyz.com\",\"content-type\":\"application/json\",\"xyz-hyt-hashregistrationid\":\"61fa7e9eb9afca3c09412a10fb02986d\",\"accept-language\":\"en-us\",\"xyz-authenticationlevel\":\"25\",\"x-forwarded-proto\":\"https\",\"dnt\":\"1\",\"x-forwarded-for\":\"173.66.169.203\",\"xyz-requestbaseurl\":\"https://offline.xyz.com/api\",\"accept\":\"*/*\",\"xyz-hyt-registrationid\":\"123456\",\"x-forwarded-server\":\"148.112.147.17\",\"singularityheader\":\"appId=7*ctrlguid=123456*acctguid=3f5d5ee2-41b3-4e59-ad10-de0b93044*ts=1498660534590*btid=2206*id=9497e8be-3275-417b-9893-30dbcca2d0*exitid=2*unresolvedexitid=164*cidfrom=148*etypeorder=CUSTOM*esubtype=CUSTOM*cidto=545\",\"xyz-hyt-sessionid\":\"2017-06-28-10:31:01-085\",\"xyz-isproxyuser\":\"false\",\"xyz-isemployee\":\"false\",\"xyz-hyt-entrypoint\":\"Mens\",\"accept-encoding\":\"gzip,deflate\",\"user-agent\":\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/603.2.5 (KHTML, like Gecko) Version/10.1.1 Safari/603.2.5\",\"xyz-iscallersecure\":\"true\"},\"correlationId\":\"bbddfa29-09fd-45c0-b136-fe417ecbb681\"}"

| rename COMMENT AS "Everything above generates sample event data; everything below is your solution"

| rex max_match=0 "(?<=,|{)\"(?<KVPcombo>[^\"]+\":\"[^\"]+)\"(?=,|})"
| streamstats count AS _serial
| mvexpand KVPcombo
| rex field=KVPcombo "^(?<KVPkey>[^\"]+)\":\"(?<KVPvalue>.*)$"
| eval {KVPkey}=KVPvalue
| fields - KVP*
| stats values(*) AS * BY _time _raw _serial

[gcusello]

Hi newbie2tech,

I'm not sure to have understood your need, try something like this

\"(?<my_field1>xyz-[^\"]*)\":\"(?<my_field2>[^\"]*)

You can test it at https://regex101.com/r/mmPWQf/1
Bye.
Giuseppe