topic Re: How to regex multiple events, store it in one variable and display based on User click? in Splunk Search

How to regex multiple events, store it in one variable and display based on User click?

rkrish71 — Thu, 12 Mar 2020 23:09:58 GMT

Hi,

I am looking for some help on the below query. I have list of APIs which has different parameters in the URL. I have extracted the Values from the URL and stored it in a variable using replace command.

Question:
1) How would I be able to combine them and store it in one Regex variable?
2) If I had it stored in one variable, will it be possible to display the count based on the selected api?

Splunk Query:

index=abcd appname=xyz
| rex field=message "(GET|POST).(?[^\?\s]+)"
| rex field=message "HTTP\/\S+.(?[^\ ]+)"
| search RespCode=50*
| eval api=replace(api, "(/api/abc/v2/user/Id/.*)","/api/abc/v2/user/Id/Unique_Value")
| eval api=replace(api, "(/api/abc/v2/Name/.*)","/api/abc/v2/user/Name/Unique_Value")
| eval api=replace(api, "(/api/abc/v2/user/.*)","/api/abc/v2/user/Unique_Value")
| eval api=replace(api, "(/api/abc/v2/name/.*/info)","/api/abc/v2/name/unique_value/info")
| eval api=replace(api, "(/api/abc/v2/info/.*/name)","/api/abc/v2/info/unique_value/name")
| rex field=message "user.Id.(?[^\ ]+)"
| stats count

Can someone help if there is a scope within splunk queries to solve this? I am still trying to learn. Appreciate any assistance. Thank you.

Re: How to regex multiple events, store it in one variable and display based on User click?

richgalloway — Fri, 13 Mar 2020 00:10:49 GMT

Something may have been lost from your regular expressions. Please edit your question to restore them so we can better understand what you are doing. It seems you only have 2 fields (variables) - api and message. How do you want to combine them?

Re: How to regex multiple events, store it in one variable and display based on User click?

rkrish71 — Fri, 13 Mar 2020 00:19:10 GMT

Hi,
Thanks for your response and pointing out about the regex. Yes it got missed out the variable names. I tried to update it in the original question, but I couldn't so reposting the question with the regex query update below.

Question with update:
Hi,

Question:
1) How would I be able to combine them and store it in one Regex variable?
2) If I had it stored in one variable, will it be possible to display the count based on the API? like lets say I display the count in a table and have another panel in drilldown that displays when the table is clicked the count.

Splunk Query:

index=abcd appname=xyz
| rex field=message "(GET|POST).(?<api>[^\ ]+)"
| rex field=message "HTTP\/\S+.(?<RespCode>[^\ ]+)"
| search RespCode=50*
| eval api=replace(api, "(/api/abc/v2/user/Id/.*)","/api/abc/v2/user/Id/Unique_Value")
| eval api=replace(api, "(/api/abc/v2/Name/.*)","/api/abc/v2/user/Name/Unique_Value")
| eval api=replace(api, "(/api/abc/v2/user/.*)","/api/abc/v2/user/Unique_Value")
| eval api=replace(api, "(/api/abc/v2/name/.*/info)","/api/abc/v2/name/unique_value/info")
| eval api=replace(api, "(/api/abc/v2/info/.*/name)","/api/abc/v2/info/unique_value/name")
| rex field=message "user.Id.(<ID>?[^\ ]+)"
| stats dc(ID)

Can someone help if there is a scope within splunk queries to solve this? I am still trying to learn. Appreciate any assistance. Thank you.

Re: How to regex multiple events, store it in one variable and display based on User click?

richgalloway — Fri, 13 Mar 2020 12:49:32 GMT

So you already have the api value in a single field called 'api'. What do you want to do with it?
Be aware that the stats command discards all fields except for "dc(ID)" so any attempt to access 'api' will return no results.
Please tell us more about what you want your results to look like.

Re: How to regex multiple events, store it in one variable and display based on User click?

rkrish71 — Fri, 13 Mar 2020 23:32:40 GMT

Unique value is the variable i am storing record/entry of any info that's not static but generated different each time.

the regular expression that has the value variable "ID" stores only one pattern. (i.e) the first one (/api/abc/v2/user/Id/Unique_Value). So, i want to do the same for all the other different apis and store it in the same "ID" variable instead of creating different regular expressions for each api and store it in different variables.

Once I store this in one variable "ID". I am looking to have this in a drilldown and depending upon which api someone clicks, it should give the unique value count of that.

Re: How to regex multiple events, store it in one variable and display based on User click?

to4kawa — Sat, 14 Mar 2020 05:53:12 GMT

|makeresults
| eval _raw="/api/abc/v2/user/Id/abcdefg
/api/abc/v2/user/Id/abcdefs
/api/abc/v2/Name/test
/api/abc/v2/Name/tests
/api/abc/v2/user/okiefshi
/api/abc/v2/user/okie3shi
/api/abc/v2/name/gift/info
/api/abc/v2/name/giftw/info
/api/abc/v2/info/nennde/name
/api/abc/v2/info/nennae/name"
|makemv delim="
" _raw
| stats count by _raw
| rename COMMENT as "this is sample, from here , the logic"
| rex  mode=sed "s/(?i)(\/api\/abc\/v2\/(user\/Id|user|name|info)\/)(?<replace_field>.*?)($|\/(info|name))/\1Unique_value\4/"
| stats count by _raw

Re: How to regex multiple events, store it in one variable and display based on User click?

woodcock — Sat, 14 Mar 2020 17:34:58 GMT

I would put your stuff in a Lookup File named URL2API.csv like this:

URL,API
/api/abc/v2/user/Id/.*,/api/abc/v2/user/Id/Unique_Value
/api/abc/v2/Name/.*,/api/abc/v2/user/Name/Unique_Value
/api/abc/v2/user/.*,/api/abc/v2/user/Unique_Value
/api/abc/v2/name/.*/info,/api/abc/v2/name/unique_value/info
/api/abc/v2/info/.*/name,/api/abc/v2/info/unique_value/name

Then create a Lookup Definition called URL2API with WILDCARD(URL)
Then you can do this:

... | lookup URL2API URL AS api OUTPUT API

If you create automatic Field Extractions for api then you can create an Automatic Lookup so that you will always have api and API whenever you search this sourcetype. Whenever you need to update the mapping, just adjust the Lookup File and everything else will instantly adjust.

Re: How to regex multiple events, store it in one variable and display based on User click?

rkrish71 — Thu, 26 Mar 2020 05:57:00 GMT

Thanks. I tried the above but the issue is, the unique values differ and there are 100s of unique values. I cannot list all of them in raw as each time it varies. Can you suggest how to tackle that?

Re: How to regex multiple events, store it in one variable and display based on User click?

to4kawa — Thu, 26 Mar 2020 09:37:25 GMT

100s of unique values
where?
If these are at same position, you can make
Regex