topic Re: extract fields from rex _raw in Splunk Search

extract fields from rex _raw

oustinov — Wed, 18 Apr 2018 15:29:25 GMT

trying to extract a fields from logfile's text (have both examples in logfile):

search sourcetype=apache "/apps/public/client1/local/" | "rex field=_raw "/apps/public/(?<client>\w+)/(?<region>\w+)/"" - works perfect, i see a fields "client" and "region" with correct client names

search sourcetype=apache "/apps/public/v4/client1/local/" | "rex field=_raw "/apps/public/v4/(?<client>\w+)/(?<region>\w+)/"" - does not work - no fields "client" and "region"

why and what is wrong?

Re: extract fields from rex _raw

elliotproebstel — Wed, 18 Apr 2018 16:54:52 GMT

Can you edit this and wrap the middle two lines with the code button (or single backticks, if you can't get the code button to work for you)? The formatter is eating some of your rex, which makes it harder to diagnose!

Re: extract fields from rex _raw

FrankVl — Wed, 18 Apr 2018 17:43:22 GMT

Are you very sure this is how you entered them? What are those " doing before rex and at the end of the line? Are you sure you actually typed those into the search bar for both attempts?

Re: extract fields from rex _raw

niketn — Wed, 18 Apr 2018 18:08:05 GMT

@oustinov, I am surprised as to how your first code is working. Ideally you are supposed to escape backslashes in regular expression with a forward slash. Following is how your second query should look like. You should also try to test regular expressions on regex101.com

search sourcetype=apache "/apps/public/v4/client1/local/"
| rex "\/apps\/public\/v4\/(?<client>\w+)\/(?<region>\w+)\/"

Try the following run anywhere search for testing:

| makeresults
| eval _raw="/apps/public/v4/client1/local/"
| rex "\/apps\/public\/v4\/(?<client>\w+)\/(?<region>\w+)\/"

Re: extract fields from rex _raw

oustinov — Wed, 18 Apr 2018 20:37:09 GMT

sorry " before rex is a typo

Re: extract fields from rex _raw

oustinov — Wed, 18 Apr 2018 20:50:40 GMT

Hello,
thanks for answer, but... 1st (without "/v4/") works in both variants, 2nd - same result - no fields extracted 😞

search sourcetype=apache  "/apps/public/v4/*" | rex field=_raw "/apps/public/v4/(?<client>\w+)/(?<region>\w+)/"

It finds all entries with "/apps/public/v4/" but does not extract fields at all. by delete /v4/ - it put "v4" as a client name 😞
does splunk have any restrictions on "v4"? 🙂

Re: extract fields from rex _raw

niketn — Thu, 19 Apr 2018 03:29:03 GMT

Did you check out the run anywhere search. If the run anywhere search is working but not with your raw data may be the pattern is different than what you have provided. Add some sample of raw events in the case. Mock/anonymize any sensitive data from the event keeping the pattern similar to what is present.

Does both URL v4 and non-v4 end with region? or is there any content after region. Is the URL followed by Space or Double Quotes or any other pattern?

Please add some samples.

Since the provided answer did not work as expected I have converted my answer to comment. So that other community experts can also look at this unanswered question and reply.

Re: extract fields from rex _raw

p_gurav — Thu, 19 Apr 2018 04:13:39 GMT

Can you try below regex:

\/apps\/public\/*\w*\d*\/(?<client>\w+)\/(?<region>\w+)\/

Re: extract fields from rex _raw

oustinov — Thu, 19 Apr 2018 10:39:37 GMT

it does not work - it take v4 as a client 😞

Re: extract fields from rex _raw

elliotproebstel — Thu, 19 Apr 2018 14:22:56 GMT

Give this a shot:

search sourcetype=apache "/apps/public/client1/local/" OR "/apps/public/v4/client1/local/"
| rex "\/apps\/public(\/v4)?\/(?<client>[^\/]+)\/(?<region>[^\/]+)"

This should ignore v4 if present but still allow it to work on URLs that do not contain v4. If you need it more generic, please share some examples where it fails, and I can help you iterate.

Re: extract fields from rex _raw

oustinov — Thu, 19 Apr 2018 15:28:30 GMT

Exact your rex does not work, but this:

rex field=_raw "\/apps\/public(\/v4)?\/(?<client>\w+)\/(?<region>\w+)"

works perfect!
Many thanks!!!

Re: extract fields from rex _raw

oustinov — Fri, 20 Apr 2018 07:20:09 GMT

Hi,
this does not work, please see answer above - this works 100%
thanks!