topic Re: How do I use regex to extract URL parameter field names in Splunk Search

How do I use regex to extract URL parameter field names

DPWSplunkPOC — Thu, 07 Apr 2016 20:19:21 GMT

I want to extract the field names from a URL's parameters. For example my raw event might look like this:

action=accept host=myserver timestamp=01/01/2016:12:00:00 src_ip=1.1.1.1 domain=mydomain process=GET url=mywebpage.com/requestedpage.resquest?field1=value1&field2=value2&field3=value3

I would like the regex to capture all field names from the parameters into one capture group called url_parameter.

I have the following regex:

^[^\?\n]\*\?(?P<\url_parameter\>\w+)*(?:[^&\n]\*&(\w+))

*Note I added slashes around url_parameter because it looked like an HTML tag.

It captures the first field after the question mark and places it in url_parameter capture group. It captures the the second field in a different capture group. Finally it does not capture field 3 or any number of remaining fields in the parameter. I'm far from a regex expert but I'm trying to teach myself. Any help is appreciated. Thank you in advance.

Re: How do I use regex to extract URL parameter field names

somesoni2 — Thu, 07 Apr 2016 20:24:12 GMT

Try something like this

your base search | rex  "^[^\?\n]*\?(?P<url_parameter>\S+)"  | rex max_match=0 field=url_parameter "(?<url_parameter>\w+)="

Re: How do I use regex to extract URL parameter field names

javiergn — Thu, 07 Apr 2016 20:30:05 GMT

Hi,

Try this:

your search here
| rex max_match=0 "(?msi)(?<url_parameter>field\d)"

It'll create a multvalue field with all your field names that you can later on used the way you like.

Re: How do I use regex to extract URL parameter field names

DPWSplunkPOC — Thu, 07 Apr 2016 20:36:12 GMT

That works! How could I make so that the user could search url_parameters without typing the rex command? Can this be added to props.conf or transforms.conf?

In the meantime you have given me a great head start!

Re: How do I use regex to extract URL parameter field names

somesoni2 — Thu, 07 Apr 2016 20:45:57 GMT

See this Splunk documentation page
http://docs.splunk.com/Documentation/Splunk/6.0.3/Knowledge/Createandmaintainsearch-timefieldextractionsthroughconfigurationfiles#Handling_events_with_multivalue_fields

Props.conf

[yoursourcetype]
REPORT-urlparams = geturlparams

Transforms.conf

[geturlparams]
REGEX = (?<url_parameter>\w+)=
MV_ADD = true