topic Re: Head scratcher regex in Splunk Search

Head scratcher regex

dbcase — Fri, 26 Jan 2018 16:24:34 GMT

I have the below data and need to extract three things, 2 of which are pretty easy (method (GET or POST) and responseStatus (numeric value), those I can do). The one that I'm having trouble with is extracting the last segment of the URL. For example in the first line I'd like to extract obtainToken. On the second line address. The third line roost and the fourth line getAllLightingStatus. I've tried several variations to partial success but not all success.

{"method":"POST","url":"/rest/blah/sites/1004057/obtainToken?tokenType=WS_CVR","params":{},"requestStartTime":1516982978230,"responseStatus":200,"responseStatusText":"OK","success":true,"responseTime":1516982979338} 

{"method":"GET","url":"/rest/blah/sites/1004057/address","params":{},"requestStartTime":1516982978142,"responseStatus":200,"responseStatusText":"OK","success":true,"responseTime":1516982978901}   

{"method":"POST","url":"/rest/blah/sites/1004057/cloudIntegrations/roost","params":{"method":"POST","path":"/iCtrlGetDeviceStatus"},"requestStartTime":1516982978032,"responseStatus":200,"responseStatusText":"OK","success":true,"responseTime":1516982979118}    

{"method":"GET","url":"/rest/blah/sites/1004057/network/lights/getAllLightingStatus","params":{},"requestStartTime":1516982978146,"responseStatus":500,"responseStatusText":"Internal Server Error","success":false,"responseTime":1516982978914,"data":"Device not connected to server"}

Re: Head scratcher regex

elliotproebstel — Fri, 26 Jan 2018 16:42:56 GMT

How about this for getting the final_segment of the url:
"url":"[^"]+\/(?<final_segment>\w+)(\?[^\/"]+)?"

Re: Head scratcher regex

cpetterborg — Fri, 26 Jan 2018 16:54:15 GMT

Just to account for potentially other characters besides \w in the field and potentially a blank field:

"url":"[^"]+\/(?<final_segment>[^"?]*)(\?[^\/"]+)?"

It's also about 20% more efficient.

Re: Head scratcher regex

elliotproebstel — Fri, 26 Jan 2018 16:57:56 GMT

Thanks, guru! I appreciate the correction.

Re: Head scratcher regex

dbcase — Fri, 26 Jan 2018 17:31:16 GMT

Wow, I feel like a babe in the woods! Thanks guys!!!

Re: Head scratcher regex

mbenwell — Tue, 29 Sep 2020 17:51:39 GMT

Quick question, any reason you're writing regex for the other objects in that data? It looks like a json object.

With the correct sourcetype the url field should exist already, then you could use a transform like the below (Hoping for feedback on performance from @cpetterborg on the below being the master of regex 🙂 😞

[extract_page_from_url]
SOURCE_KEY = url
REGEX = \S+\/([^\/\?]+)
FORMAT = page::$1

Also, check out the URL toolbox app (https://splunkbase.splunk.com/app/2734/), very handy when working with URL's