topic Re: how to not index log lines that have these 2 phrases in them? in Getting Data In

how to not index log lines that have these 2 phrases in them?

joesrepsol — Fri, 26 May 2017 19:54:01 GMT

Looking to get the correct regex statement for my transforms.conf to select both the "(vert.x-eventloop-thread-4)" and the "Request is valid" strings. If these both exist on that same line, I do not want to ingest that line into Splunk.

And what if I want to also exclude other lines of logs that contain other text strings? Like to also exclude rows with "validating the required header fields"?

Can I combine multiple regex lines into one? Not sure how that works.

Example Log:
2017-05-25 23:22:31,422 INFO (vert.x-eventloop-thread-4) ?.: TID: b47d618c-a5fc-4d41-a7a2-b1c0fae15dc0 - Request is valid
2017-05-25 23:22:31,419 INFO (vert.x-eventloop-thread-4) ?.: TID: b47d618c-a5fc-4d41-a7a2-b1c0fae15dc0 - validating the required header fields

Thinking I got the rest...

Props.conf
[default]
TRANSFORMS-null= setnull

Transforms.conf
[setnull]
REGEX = ???HELP???
DEST_KEY = queue
FORMAT = nullQueue

Re: how to not index log lines that have these 2 phrases in them?

richgalloway — Fri, 26 May 2017 20:22:12 GMT

This regex string should get you started.

\(vert.x-eventloop-thread-4\).*(Request is valid|validating the required header fields)

Re: how to not index log lines that have these 2 phrases in them?

joesrepsol — Fri, 26 May 2017 20:24:18 GMT

Do I need the leading "\" on that regex line? So would it look like this?

[setnull]
REGEX = (vert.x-eventloop-thread-4).*(Request is valid|validating the required header fields)
DEST_KEY = queue
FORMAT = nullQueue

Thanks in advance!

Re: how to not index log lines that have these 2 phrases in them?

jwalbert — Fri, 26 May 2017 20:37:23 GMT

Adjust to your liking, I left the ending string as dot start soup (.*) for you to either leave non-specific or create additional options if required.

\(vert\.[^)]+\)(?:[^-]++|-(?=\w+))*-\s(?:.*)

This has the added benefit of using greedy negation on the hyphen with proper alternation for some regex engine optimization.

--
Joe

Re: how to not index log lines that have these 2 phrases in them?

woodcock — Fri, 26 May 2017 22:01:20 GMT

Try this:

REGEX = \s+\(vert.x-eventloop-thread-4\)\s+|\s+- Request is valid|\s+- validating the required header fields

Re: how to not index log lines that have these 2 phrases in them?

DalJeanis — Fri, 26 May 2017 22:19:58 GMT

you need the escape/slash because you are describing an actual open parenthesis character, not starting a group.

Re: how to not index log lines that have these 2 phrases in them?

joesrepsol — Fri, 26 May 2017 22:34:24 GMT

Trying all these combinations... waiting for testing to confirm which works best (or at all). THANKS FOR ALL THE SUGGESTIONS!

Re: how to not index log lines that have these 2 phrases in them?

joesrepsol — Fri, 26 May 2017 23:14:29 GMT

Looks like I no longer need to check for the two string matches per index line... now I just need to eliminate any logs that have these entries... So this got simpler, but I'm still fumbling on the REGEX line to start...

I want to eliminate any logs with any of these 3 values in that log.
"Request is valid" or "validating the required header fields" or "getSystemSetupHandler"

Tried this line, but it's only seeing the 2nd and 3rd values (which is working) but I'm still getting events with the 1st value (Request is valid).

REGEX = \s+- Request is valid|\s+- validating the required header fields|\s+- getSystemSetupHandler

What am I missing?

Re: how to not index log lines that have these 2 phrases in them?

joesrepsol — Tue, 29 Sep 2020 14:17:13 GMT

Did some testing on regex101.com and this "looks" like it works, but the events are still coming in and being ingesting into Splunk. So I'm missing something. Tried many versions of the example above, and regex101.com doesn't validate everything as it should be. Should I not use Regex101.com for splunk regex??

So out of the events coming in I don't want any with these matching phrases:
“Request is valid”
“validating the [\w-]+ header field. value:”
“Validated service ticket”
“getAppActiveAndSecretResult:”
“getSystemSetupHandler:”
“validating the required header fields.”

Example Log Data:
2017-05-25 23:22:31,422 INFO (vert.x-eventloop-thread-4) ?.: TID: b47d618c-a5fc-4d41-a7a2-b1c0fae15dc0 - Request is valid
2017-05-25 23:22:31,422 INFO (vert.x-eventloop-thread-4) ?.: TID: b47d618c-a5fc-4d41-a7a2-b1c0fae15dc0 - validating the ur-content-checksum header field. value: IRHP9g6wxkouF35KI1DUmg==
2017-05-25 23:22:31,422 INFO (vert.x-eventloop-thread-4) ?.: TID: b47d618c-a5fc-4d41-a7a2-b1c0fae15dc0 - Validated service ticket
2017-05-25 23:22:31,422 INFO (vert.x-eventloop-thread-4) ?.: TID: b47d618c-a5fc-4d41-a7a2-b1c0fae15dc0 - validating the ur-user-name header field. value: dp_prod_loader
2017-05-25 23:22:31,422 INFO (vert.x-eventloop-thread-4) ?.: TID: b47d618c-a5fc-4d41-a7a2-b1c0fae15dc0 - validating the ur-requesting-service header field. value: ur.java.client
2017-05-25 23:22:31,422 INFO (vert.x-eventloop-thread-4) ?.: TID b47d618c-a5fc-4d41-a7a2-b1c0fae15dc0 - getAppActiveAndSecretResult: 2 ms
2017-05-25 23:22:31,420 INFO (vert.x-eventloop-thread-4) ?.: TID b47d618c-a5fc-4d41-a7a2-b1c0fae15dc0 - getSystemSetupHandler: 1 ms
2017-05-25 23:22:31,419 INFO (vert.x-eventloop-thread-4) ?.: TID: b47d618c-a5fc-4d41-a7a2-b1c0fae15dc0 - validating the ur-app-id header field. value: DIGITAL_PLATFORM
2017-05-25 23:22:31,419 INFO (vert.x-eventloop-thread-4) ?.: TID: b47d618c-a5fc-4d41-a7a2-b1c0fae15dc0 - validating the ur-request-date header field. value: 2017-05-25 23:22:30.963 -0700
2017-05-25 23:22:31,419 INFO (vert.x-eventloop-thread-4) ?.: TID: b47d618c-a5fc-4d41-a7a2-b1c0fae15dc0 - validating the required header fields.

props.conf
[default]
TRANSFORMS-null= setnull

transforms.conf
[setnull]
REGEX = Request is valid|getAppActiveAndSecretResult|getSystemSetupHandler|validating the [\w-]+ header field|Validated service ticket
DEST_KEY = queue
FORMAT = nullQueue