Splunk: Extract runtime search wildcards into key value pairs for analysis

[jmascherino]

I have a log4j server log with multiple lines formatted similar to the following:

"10.1.1.1" "AUTH-USER" "22/Jul/2013:22:42:42 -0700" "GET /source1/resources/RESOURCE/ENDPOINT/1111/start HTTP/1.1" 200 4
"10.1.1.1" "auth2" "22/Jul/2013:22:43:03 -0700" "PUT /source1/resources/RESOURCE/ENDPOINT HTTP/1.1" 200 4
"10.40.16.254" "auth2" "22/Jul/2013:22:43:03 -0700" "PUT /source1/resources/secure/RESOURCE/v1/ENDPOINT?var1=A&var2=01-01-2013&var4=Allison HTTP/1.1" 200 4

Where RESOURCE is a list of variable strings and ENDPOINT represents list of variable strings.

I would like to count the number of times a distinct pair of RESOURCE:ENDPOINT exists in the log file to know the number of times each web service has been called in a specific timeline.

I know I can perform the following search to return all of the values:

("/RevWebServices/resources/*/* HTTP/1.1" OR "/RevWebServices/resources/secure/*/v*/* HTTP/1.1")

But how do I extract the values in to key value pair that can be counted?

[cramasta]

You can use the rex command to extract a new field from your data. because i dont have complete view into your data to see all the possible combinations i made this generic regex that will get your close to what you need

("/RevWebServices/resources/*/* HTTP/1.1" OR "/RevWebServices/resources/secure/*/v*/* HTTP/1.1") | rex "(?:"PUT|GET) (?<newfieldname>.*?)(?:\s|\?)"

this should extract a new field called "newfieldname" with the following values based on your examples above

/source1/resources/RESOURCE/ENDPOINT/1111/start
/source1/resources/RESOURCE/ENDPOINT
/source1/resources/secure/RESOURCE/v1/ENDPOINT

[lukejadamec]

SomeSearch | rex field=_raw "/RevWebServices/resources/(?.) HTTP/1.1.” |stats count by resource

SomeSearch | rex field=_raw "/RevWebServices/resources/secure/(?.) HTTP/1.1.” | stats count by resource_secure

You may have to modify the regex, but this is basically how you extract fields. Once you have them as a field, you can do pretty much anything with them.