topic Re: multiple occurences of one field - from rex to field-extraction in Splunk Search

multiple occurences of one field - from rex to field-extraction

Marco204 — Tue, 27 Apr 2021 12:34:19 GMT

Hi there,

so I have a search that results contains multiple occurences of one field.

My current solution is using rex together with max_match=0 in order to get this:

index="dev_logs" pod::apollo* some.url.com/api statusCode | rex field=_raw max_match=0 "\"statusCode\":(?<ApolloStatusCode>\d+)"

Well, right now I want an alert for the case that status is neither 200 nor 204.

So I played around with this:

| search 200 OR search 204 | search NOT 200 AND search NOT 204 | search NOT [search 200 OR search 204]

To be honest neither works 🤔

Right now I think that the sub-search is the problem, and a solution could be to use field-extraction.

So I used the field extraction wizard and changed the generated regex to this afterwards:

"statusCode":(?<ApolloStatusCode>\d+)

But this only returns the first occurence - but I need them all.

With field transformation I didn't make any progress, and editing some conf files are out of scope...

Thanks for any help,

Marco

Re: multiple occurences of one field - from rex to field-extraction

richgalloway — Tue, 27 Apr 2021 18:11:38 GMT

Use the field name in your search.

index="dev_logs" pod::apollo* some.url.com/api statusCode | rex field=_raw max_match=0 "\"statusCode\":(?<ApolloStatusCode>\d+)" | search NOT (ApolloStatusCode=200 OR ApolloStatusCode=204)

That's the normal case, but you're using max_match=0, which produces a multi-value field and that's not so normal.

index="dev_logs" pod::apollo* some.url.com/api statusCode | rex field=_raw max_match=0 "\"statusCode\":(?<ApolloStatusCode>\d+)" | where (isnull(mvfind(ApolloStatusCode, 200)) AND isnull(mvfind( ApolloStatusCode,204)))

Re: multiple occurences of one field - from rex to field-extraction

Marco204 — Wed, 28 Apr 2021 10:30:04 GMT

Thanks a lot for your answer. In between I was able to combine field transformation and field extraction, so right now I have the variable named MARCO. In a search over the last 30 day there occur 3 values: 200, 204 and 400. But using your where-clause returns no results at all. And the 2nd param of the mvfind function is a regex, so it has to be quoted.

Here some results:

| where isnull(mvfind(MARCO,"204"))

return the 200 and 400.

| where isnull(mvfind(MARCO,"400"))

return the 200 and 204.

| where isnull(mvfind(MARCO,"200"))

return 0 events. 🙄

So now I need to explain the results a normal search without where-clause.

200,204,200
200
200,204
200,400
200,200,204

So somehow there is always a 200, and removing those results in 0 events.

Do you know another way of filtering?

Currently I want the (200,400) and maybe in the future I will get something like (200,500) or (200,404) which I also want...

Best regards,
Marco

Re: multiple occurences of one field - from rex to field-extraction

richgalloway — Wed, 28 Apr 2021 13:03:04 GMT

Try this search for non-200 result codes.

index="dev_logs" pod::apollo* some.url.com/api statusCode | rex field=_raw max_match=0 "\"statusCode\":(?<ApolloStatusCode>\d+)" | where isnotnull(mvfind(ApolloStatusCode, "[3-9]\d{2}")