Splunk Search

Filter several strings in transforms.conf

gozulin
Communicator

EDITED to add relevant info:

I'm trying to prevent indexing of entries containing certain strings (ACDB0000,ACM0033,W0032,L0041, \[DEBUG\])

This stanza worked fine when all I wanted to filter was debug entries:

#old transforms.conf
[setnull]
REGEX = \[DEBUG\])
DEST_KEY = queue
FORMAT = nullQueue 

But when I add a few more string to the REGEX, failure ensues. The log events containing ACM0033 and ACDB0000 aren't getting filtered out. They are still getting indexed. This is the new transforms file:

#new transforms.conf
[setnull]
REGEX = (W0032|L0041|ACM0033|ACDB0000|\[DEBUG\]) 
DEST_KEY = queue
FORMAT = nullQueue

This is my props.conf for both configs:

# same props.conf
[default]
TRANSFORMS-null = setnull
CHARSET = AUTO
NO_BINARY_CHECK = 1
pulldown_type = 1

[foo-prod]
TIME_FORMAT = %b %d %H:%M:%S
NO_BINARY_CHECK = 1
pulldown_type = 1

The log entries in question look something like this. A date, a level of severity in brackets and then a string of varying length. All generated in the standard Unix syslog format

2014-05-13 22:56:20,988 [INFO] ACDB0000: ACDB_LOG - IncomingRequest. guid=AN-ON method=register idx=0 <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><ns2:register 
0 Karma

gozulin
Communicator

What is the reason for changing the format from nullQueue to indexQueue?

Also, what is the function of:

[changeme_setnull]
REGEX =  .
DEST_KEY = queue
FORMAT = nullQueue

I would appreciate it if you would explain your reasoning to me so I could learn more.

0 Karma

gozulin
Communicator

I have edited my post to answer your question. Please let me know if you'd like more info.

0 Karma

alemarzu
Motivator

props.conf

[default]
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%9N
TRANSFORMS-changeme = changeme_setnull, changeme_setparsing
CHARSET = AUTO
NO_BINARY_CHECK = 1
pulldown_type = 1

transforms.conf

[changeme_setparsing]
REGEX = (W0032|L0041|ACM0033|ACDB0000|[DEBUG])
DEST_KEY = queue
FORMAT = indexQueue

[changeme_setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

What kind of log u have there ? I'm not sure but you can try this. (kinda new to splunk so bare with me)

0 Karma

gozulin
Communicator

What is the reason for changing the format from nullQueue to indexQueue?

Also, what is the function of:

[changeme_setnull]
REGEX =  .
DEST_KEY = queue
FORMAT = nullQueue

I would appreciate it if you would explain your reasoning to me so I could learn more.

0 Karma

gozulin
Communicator

I tried this search query:
W0032 OR L0041 OR ACM0033 OR ACDB0000 OR \[DEBUG\]
It finds ACM0033 and ACDB0000 entries but DEBUG, W0032 and L0041 appear to be filtered properly.

0 Karma

alemarzu
Motivator

Am I wrong or its a multiline log ? Did u try your regex in a search query ?

btw: I've edited the transforms.conf check that out if it works.

0 Karma

alemarzu
Motivator

As Rich said, can u provide some sample ? And what was the outcome ?

0 Karma

gozulin
Communicator

they are syslog generated on linux/solaris machines

0 Karma

richgalloway
SplunkTrust
SplunkTrust

How is it not working? Filtering too much or too little? Please provide some sample log events.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...