topic Re: Why is my nullQueue configuration not working at app level? in Splunk Search

Why is my nullQueue configuration not working at app level?

vkakani60 — Thu, 14 Jul 2016 21:15:38 GMT

I would like to eliminate the unnecessary content in the events because I have a small license. I want to remove the text and 20 . characters from each of the events. So I added two stanzas in transforms.conf: one for removing the . dot, and one for removing text. Even these small stanzas make a huge impact on my very frequently rolling logs.

transforms.conf

[nullifytext]
REGEX = ^\.{20}$
DEST_KEY = queue
FORMAT = nullQueue

props.conf

[webserverIH]
TRANSFORMS-removedots= nullifytext
category = Application
description = webserver logs
pulldown_type = 1
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
disabled = false

I have edited those logs at the search app level c:\Splunk\etc\apps\search\local and restarted Splunk. Now I have deleted the old logs and reindexed the new similar logs, but still I see the text (not shown here) and dots in the events.

UPDATE

This is my custom log file looks like. We want to remove the dots only from the splunk perspective.

....................
IP: ipaddress
Date: [07/Mar/2004:16:05:49 -0800] 
Method: "GET /topicparent=Main.ConfigurationVariables HTTP/1.1"
Response: 401
Type: Info
....................

I want to remove only the . dot and Type:info from the event, not the entire event which contain the Type:info. I need Splunk not to index those dots and test for every event, and desired output should be

IP: ipaddress
Date: [07/Mar/2004:16:05:49 -0800] 
Method: "GET /topicparent=Main.ConfigurationVariables HTTP/1.1"
Response: 401

Re: Why is my nullQueue configuration not working at app level?

sundareshr — Thu, 14 Jul 2016 21:44:24 GMT

If you trying to get rid of the entire event if it contains 20 dots, see if this (remember to restart)

 [nullifytext]
 REGEX = (\.{20})
 DEST_KEY = queue
 FORMAT = nullQueue

Don't see anything to remove text

Re: Why is my nullQueue configuration not working at app level?

rsennett_splunk — Thu, 14 Jul 2016 21:59:26 GMT

Actually, I think what you're looking for is the SEDCMD
The best example is here

nullQueue is to send and entire event (like a header that ends up being sliced into it's own event unlike all the others) to oblivion.

in this case, you really want to replace a pattern, with nothing.
this is done in the props.conf

SEDCMD-dots = s/^\.{20}//g
SEDCMD-txt = s/Type:\s+Info//g

that will replace exactly twenty periods that appear at the beginning of a string, and the "Type: Info"

it's not as slick but if I know exactly what I want to delete... there is no need for partial info in the regex. The rule of thumb is to never send Splunk running around looking for the end of the line when you can just say "stop here... you got it."

Re: Why is my nullQueue configuration not working at app level?

rsennett_splunk — Thu, 14 Jul 2016 22:01:00 GMT

SEDCMD removes text. You also don't need the capturing group. I think the problem is twofold. not needing to remove the event and the regex is perhaps wrong. stay tuned! 🙂

Re: Why is my nullQueue configuration not working at app level?

vkakani60 — Thu, 14 Jul 2016 22:21:56 GMT

Please check my question updated. Apologies for confusion.

Re: Why is my nullQueue configuration not working at app level?

vkakani60 — Thu, 14 Jul 2016 22:22:09 GMT

Please check my question updated. Apologies for the confusion.

Re: Why is my nullQueue configuration not working at app level?

sundareshr — Thu, 14 Jul 2016 22:38:34 GMT

Try SEDCMD

SEDCMD-rd=s/(\.+)//g
SEDCMD-ti=s/(Type:.*)//g

Re: Why is my nullQueue configuration not working at app level?

woodcock — Fri, 15 Jul 2016 21:34:52 GMT

I don't get it. Please confirm/correct this guess at a translation.

1: Your  7-line example is to be considered a single event across multiple lines. 
2: These events are to be dropped completely.
3: These events can be identified by ANY of:
   a: A first line starting with 20 periods.
   b: A last line starting with 20 periods.
   c: A next-to-last line that has the text "Type: Info".

Re: Why is my nullQueue configuration not working at app level?

vkakani60 — Fri, 15 Jul 2016 22:14:24 GMT

I want to get rid of three line from my seven lines code.
1: get rid of 20 dots first and last
2: get rid of type:Info line

Re: Why is my nullQueue configuration not working at app level?

woodcock — Fri, 15 Jul 2016 22:39:13 GMT

You need to use SEDCMD like this:

SEDCMD-Periods = s/^[\s\.]*[\r\n]*//g
SEDCMD-TypeInfo= s/^\s*Type:\s+Info\s*[\r\n]*//

Put this on your Indexers, restart all splunk instances there and then ALL NEW DATA (old data will remain bloated) will be truncated.

Re: Why is my nullQueue configuration not working at app level?

rsennett_splunk — Mon, 18 Jul 2016 19:25:12 GMT

did this example (or woodcock's example with a fancier regex) not work?

If they didn't work... what was the outcome?

Re: Why is my nullQueue configuration not working at app level?

vkakani60 — Thu, 11 Aug 2016 22:46:24 GMT

I have tried yours and it works as expected.

Re: Why is my nullQueue configuration not working at app level?

rsennett_splunk — Sun, 14 Aug 2016 00:55:55 GMT

Great! Glad it worked for you.