Splunk Search
Highlighted

Filter several strings in transforms.conf

Communicator

EDITED to add relevant info:

I'm trying to prevent indexing of entries containing certain strings (ACDB0000,ACM0033,W0032,L0041, \[DEBUG\])

This stanza worked fine when all I wanted to filter was debug entries:

#old transforms.conf
[setnull]
REGEX = \[DEBUG\])
DEST_KEY = queue
FORMAT = nullQueue 

But when I add a few more string to the REGEX, failure ensues. The log events containing ACM0033 and ACDB0000 aren't getting filtered out. They are still getting indexed. This is the new transforms file:

#new transforms.conf
[setnull]
REGEX = (W0032|L0041|ACM0033|ACDB0000|\[DEBUG\]) 
DEST_KEY = queue
FORMAT = nullQueue

This is my props.conf for both configs:

# same props.conf
[default]
TRANSFORMS-null = setnull
CHARSET = AUTO
NO_BINARY_CHECK = 1
pulldown_type = 1

[foo-prod]
TIME_FORMAT = %b %d %H:%M:%S
NO_BINARY_CHECK = 1
pulldown_type = 1

The log entries in question look something like this. A date, a level of severity in brackets and then a string of varying length. All generated in the standard Unix syslog format

2014-05-13 22:56:20,988 [INFO] ACDB0000: ACDB_LOG - IncomingRequest. guid=AN-ON method=register idx=0 <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><ns2:register 
0 Karma
Highlighted

Re: Filter several strings in transforms.conf

SplunkTrust
SplunkTrust

How is it not working? Filtering too much or too little? Please provide some sample log events.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: Filter several strings in transforms.conf

Motivator

props.conf

[default]
TIMEFORMAT = %Y-%m-%d %H:%M:%S,%9N
TRANSFORMS-changeme = changeme
setnull, changemesetparsing
CHARSET = AUTO
NO
BINARYCHECK = 1
pulldown
type = 1

transforms.conf

[changemesetparsing]
REGEX = (W0032|L0041|ACM0033|ACDB0000|[DEBUG])
DEST
KEY = queue
FORMAT = indexQueue

[changemesetnull]
REGEX = .
DEST
KEY = queue
FORMAT = nullQueue

What kind of log u have there ? I'm not sure but you can try this. (kinda new to splunk so bare with me)

0 Karma
Highlighted

Re: Filter several strings in transforms.conf

Communicator

they are syslog generated on linux/solaris machines

0 Karma
Highlighted

Re: Filter several strings in transforms.conf

Motivator

As Rich said, can u provide some sample ? And what was the outcome ?

0 Karma
Highlighted

Re: Filter several strings in transforms.conf

Motivator

Am I wrong or its a multiline log ? Did u try your regex in a search query ?

btw: I've edited the transforms.conf check that out if it works.

0 Karma
Highlighted

Re: Filter several strings in transforms.conf

Communicator

I tried this search query:
W0032 OR L0041 OR ACM0033 OR ACDB0000 OR \[DEBUG\]
It finds ACM0033 and ACDB0000 entries but DEBUG, W0032 and L0041 appear to be filtered properly.

0 Karma
Highlighted

Re: Filter several strings in transforms.conf

Communicator

What is the reason for changing the format from nullQueue to indexQueue?

Also, what is the function of:

[changeme_setnull]
REGEX =  .
DEST_KEY = queue
FORMAT = nullQueue

I would appreciate it if you would explain your reasoning to me so I could learn more.

0 Karma
Highlighted

Re: Filter several strings in transforms.conf

Communicator

I have edited my post to answer your question. Please let me know if you'd like more info.

0 Karma
Highlighted

Re: Filter several strings in transforms.conf

Communicator

What is the reason for changing the format from nullQueue to indexQueue?

Also, what is the function of:

[changeme_setnull]
REGEX =  .
DEST_KEY = queue
FORMAT = nullQueue

I would appreciate it if you would explain your reasoning to me so I could learn more.

0 Karma