topic Re: Filter several strings in transforms.conf in Splunk Search

Filter several strings in transforms.conf

gozulin — Tue, 13 May 2014 17:58:13 GMT

EDITED to add relevant info:

I'm trying to prevent indexing of entries containing certain strings (ACDB0000,ACM0033,W0032,L0041, \[DEBUG\])

This stanza worked fine when all I wanted to filter was debug entries:

#old transforms.conf
[setnull]
REGEX = \[DEBUG\])
DEST_KEY = queue
FORMAT = nullQueue

But when I add a few more string to the REGEX, failure ensues. The log events containing ACM0033 and ACDB0000 aren't getting filtered out. They are still getting indexed. This is the new transforms file:

#new transforms.conf
[setnull]
REGEX = (W0032|L0041|ACM0033|ACDB0000|\[DEBUG\]) 
DEST_KEY = queue
FORMAT = nullQueue

This is my props.conf for both configs:

# same props.conf
[default]
TRANSFORMS-null = setnull
CHARSET = AUTO
NO_BINARY_CHECK = 1
pulldown_type = 1

[foo-prod]
TIME_FORMAT = %b %d %H:%M:%S
NO_BINARY_CHECK = 1
pulldown_type = 1

The log entries in question look something like this. A date, a level of severity in brackets and then a string of varying length. All generated in the standard Unix syslog format

2014-05-13 22:56:20,988 [INFO] ACDB0000: ACDB_LOG - IncomingRequest. guid=AN-ON method=register idx=0 <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><ns2:register

Re: Filter several strings in transforms.conf

richgalloway — Tue, 13 May 2014 18:01:05 GMT

How is it not working? Filtering too much or too little? Please provide some sample log events.

Re: Filter several strings in transforms.conf

alemarzu — Mon, 28 Sep 2020 16:36:04 GMT

props.conf

[default]
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%9N
TRANSFORMS-changeme = changeme_setnull, changeme_setparsing
CHARSET = AUTO
NO_BINARY_CHECK = 1
pulldown_type = 1

transforms.conf

[changeme_setparsing]
REGEX = (W0032|L0041|ACM0033|ACDB0000|[DEBUG])
DEST_KEY = queue
FORMAT = indexQueue

[changeme_setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

What kind of log u have there ? I'm not sure but you can try this. (kinda new to splunk so bare with me)

Re: Filter several strings in transforms.conf

gozulin — Tue, 13 May 2014 19:16:45 GMT

they are syslog generated on linux/solaris machines

Re: Filter several strings in transforms.conf

alemarzu — Tue, 13 May 2014 19:34:41 GMT

As Rich said, can u provide some sample ? And what was the outcome ?

Re: Filter several strings in transforms.conf

alemarzu — Wed, 14 May 2014 12:27:53 GMT

Am I wrong or its a multiline log ? Did u try your regex in a search query ?

btw: I've edited the transforms.conf check that out if it works.

Re: Filter several strings in transforms.conf

gozulin — Fri, 16 May 2014 17:49:41 GMT

I tried this search query:
W0032 OR L0041 OR ACM0033 OR ACDB0000 OR \[DEBUG\]
It finds ACM0033 and ACDB0000 entries but DEBUG, W0032 and L0041 appear to be filtered properly.

Re: Filter several strings in transforms.conf

gozulin — Fri, 16 May 2014 18:17:05 GMT

I have edited my post to answer your question. Please let me know if you'd like more info.

Re: Filter several strings in transforms.conf

gozulin — Fri, 16 May 2014 19:14:54 GMT

What is the reason for changing the format from nullQueue to indexQueue?

Also, what is the function of:

[changeme_setnull]
REGEX =  .
DEST_KEY = queue
FORMAT = nullQueue

I would appreciate it if you would explain your reasoning to me so I could learn more.

Re: Filter several strings in transforms.conf

gozulin — Fri, 16 May 2014 19:15:03 GMT

What is the reason for changing the format from nullQueue to indexQueue?

Also, what is the function of:

[changeme_setnull]
REGEX =  .
DEST_KEY = queue
FORMAT = nullQueue

I would appreciate it if you would explain your reasoning to me so I could learn more.