Splunk Search
Highlighted

Why is my nullQueue configuration not working at app level?

Path Finder

I would like to eliminate the unnecessary content in the events because I have a small license. I want to remove the text and 20 . characters from each of the events. So I added two stanzas in transforms.conf: one for removing the . dot, and one for removing text. Even these small stanzas make a huge impact on my very frequently rolling logs.

transforms.conf

[nullifytext]
REGEX = ^\.{20}$
DEST_KEY = queue
FORMAT = nullQueue

props.conf

[webserverIH]
TRANSFORMS-removedots= nullifytext
category = Application
description = webserver logs
pulldown_type = 1
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
disabled = false

I have edited those logs at the search app level c:\Splunk\etc\apps\search\local and restarted Splunk. Now I have deleted the old logs and reindexed the new similar logs, but still I see the text (not shown here) and dots in the events.

UPDATE

This is my custom log file looks like. We want to remove the dots only from the splunk perspective.

....................
IP: ipaddress
Date: [07/Mar/2004:16:05:49 -0800] 
Method: "GET /topicparent=Main.ConfigurationVariables HTTP/1.1"
Response: 401
Type: Info
....................

I want to remove only the . dot and Type:info from the event, not the entire event which contain the Type:info. I need Splunk not to index those dots and test for every event, and desired output should be

IP: ipaddress
Date: [07/Mar/2004:16:05:49 -0800] 
Method: "GET /topicparent=Main.ConfigurationVariables HTTP/1.1"
Response: 401
0 Karma
Highlighted

Re: Why is my nullQueue configuration not working at app level?

Legend

If you trying to get rid of the entire event if it contains 20 dots, see if this (remember to restart)

 [nullifytext]
 REGEX = (\.{20})
 DEST_KEY = queue
 FORMAT = nullQueue

Don't see anything to remove text

0 Karma
Highlighted

Re: Why is my nullQueue configuration not working at app level?

Splunk Employee
Splunk Employee

SEDCMD removes text. You also don't need the capturing group. I think the problem is twofold. not needing to remove the event and the regex is perhaps wrong. stay tuned! 🙂

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma
Highlighted

Re: Why is my nullQueue configuration not working at app level?

Path Finder

Please check my question updated. Apologies for the confusion.

0 Karma
Highlighted

Re: Why is my nullQueue configuration not working at app level?

Legend

Try SEDCMD

SEDCMD-rd=s/(\.+)//g
SEDCMD-ti=s/(Type:.*)//g
0 Karma
Highlighted

Re: Why is my nullQueue configuration not working at app level?

Splunk Employee
Splunk Employee

Actually, I think what you're looking for is the SEDCMD
The best example is here

nullQueue is to send and entire event (like a header that ends up being sliced into it's own event unlike all the others) to oblivion.

in this case, you really want to replace a pattern, with nothing.
this is done in the props.conf

SEDCMD-dots = s/^\.{20}//g
SEDCMD-txt = s/Type:\s+Info//g

that will replace exactly twenty periods that appear at the beginning of a string, and the "Type: Info"

it's not as slick but if I know exactly what I want to delete... there is no need for partial info in the regex. The rule of thumb is to never send Splunk running around looking for the end of the line when you can just say "stop here... you got it."

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!

View solution in original post

Highlighted

Re: Why is my nullQueue configuration not working at app level?

Path Finder

Please check my question updated. Apologies for confusion.

0 Karma
Highlighted

Re: Why is my nullQueue configuration not working at app level?

Splunk Employee
Splunk Employee

did this example (or woodcock's example with a fancier regex) not work?

If they didn't work... what was the outcome?

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma
Highlighted

Re: Why is my nullQueue configuration not working at app level?

Path Finder

I have tried yours and it works as expected.

0 Karma
Highlighted

Re: Why is my nullQueue configuration not working at app level?

Splunk Employee
Splunk Employee

Great! Glad it worked for you.

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma