topic Re: Filtering logs not working as expected in Getting Data In

Filtering logs not working as expected

vimselva — Fri, 05 Jul 2024 09:00:48 GMT

I have tried to solve this problem with all the combinations, but missing some key thing on how to resolve.

I have various logs coming with source pattern as /var/log/containers/*. I would like to drop the DEBUG logs and hence have the following in props.conf:

[source://var/log/containers/*]
TRANSFORMS-null = debug_to_null

and in transforms.conf:

[debug_to_null]
REGEX = DEBUG
DEST_KEY = queue
FORMAT = nullQueue

After making the above change, as expected the logs with DEBUG keyword is getting dropped.

Now, I would also like to drop logs with another pattern for a particular source pattern under /var/log/containers, so I've updated my props.conf like this:

[source::/var/log/containers/*_integration-business*.log]
TRANSFORMS-null = setnull

[source://var/log/containers/*]
TRANSFORMS-null = debug_to_null

and updated transforms.conf like this:

[debug_to_null]
REGEX = DEBUG
DEST_KEY = queue
FORMAT = nullQueue

[setnull]
REGEX = NormalizedApiId failed to resolve
DEST_KEY = queue
FORMAT = nullQueue

After making this change, I can see only logs with DEBUG keyword is getting dropped, however the logs with NormalizedApiId failed to resolve are still being ingested.

I was hoping that logs with DEBUG keyword from all source paths with /var/log/containers/* pattern will be dropped and NormalizedApiId failed to resolve keyword from a particular source path with /var/log/containers/*_integration-business*.log pattern will be dropped. But seems not working that way.

Please guide me on this.

Re: Filtering logs not working as expected

gcusello — Fri, 05 Jul 2024 09:22:41 GMT

Hi @vimselva ,

check in search dashboard if the regex is correct using the regex command:

<your_search> | regex "NormalizedApiId failed to resolve"

Ciao.

Giuseppe

Re: Filtering logs not working as expected

isoutamo — Fri, 05 Jul 2024 15:09:28 GMT

try to change another TRANSFORMS definition to something else like TRANSFORMS-null-ib-log instead of use just TRANSFORS-null twice.

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Attributeprecedencewithinafile

r. Ismo

Re: Filtering logs not working as expected

vimselva — Sat, 06 Jul 2024 01:35:34 GMT

Thank you @isoutamo , that worked after updating the key.

However, still puzzled on after adding another stanza. My current props.conf:

[source::/var/log/containers/*_integration-business*.log]
TRANSFORMS-int-null = setnull

[source::/var/log/containers/*_0091*.log]
TRANSFORMS-set = allow_all

[source::/var/log/containers/*]
TRANSFORMS-null = debug_to_null

and transforms.conf:

[allow_all]
REGEX = .
DEST_KEY = QUEUE
FORMAT = indexQueue

[setnull]
REGEX = NormalizedApiId failed to resolve
DEST_KEY = queue
FORMAT = nullQueue

[debug_to_null]
REGEX = DEBUG|Debug|debug
DEST_KEY = queue
FORMAT = nullQueue

So I would like to allow any logs including DEBUG keyword from the source pattern /var/log/containers/*_0091*.log, however I see the logs with DEBUG keyword from that particular source pattern is being dropped. Is this even possible to do?

Re: Filtering logs not working as expected

vimselva — Sat, 06 Jul 2024 01:36:12 GMT

Yes the regex is correct and validated.

Re: Filtering logs not working as expected

isoutamo — Sat, 06 Jul 2024 14:21:01 GMT

Maybe you could utilize that priority attribute with those two sources and use same TRANSFORMS-null attribute with both of those sources? See details from previous doc link.

Re: Filtering logs not working as expected

vimselva — Sun, 07 Jul 2024 11:13:30 GMT

You are great, that worked! Thank you for sharing knowledge.