topic Re: Routing to nullQueue - version 6.2 in Getting Data In

Routing to nullQueue - version 6.2

pattypayscale — Tue, 29 Sep 2020 07:19:58 GMT

Hello All,
I am attempting to filter out specific events from a given input, they're useless and I don't want to waste resources indexing them. I have found a number of seemingly great answers and help pages for how to set this up with a props.conf and transforms.conf. Below are the dumps from btool for my app's inputs.conf, props.conf and transforms.conf:

inputs.conf:

C:\Program Files\Splunk\etc\apps\CDN_MKXLVI\local\inputs.conf [aws_s3://cdn-mkXLVI]
C:\Program Files\Splunk\etc\apps\CDN_MKXLVI\local\inputs.conf aws_account = SplunkIAM
C:\Program Files\Splunk\etc\apps\CDN_MKXLVI\local\inputs.conf bucket_name = testbucket
C:\Program Files\Splunk\etc\apps\CDN_MKXLVI\local\inputs.conf key_name = cflogs
C:\Program Files\Splunk\etc\apps\CDN_MKXLVI\local\inputs.conf sourcetype = mkXLVI:aws:s3

props.conf:

C:\Program Files\Splunk\etc\apps\CDN_MKXLVI\local\props.conf [sourcetype::mkXLVI:aws:s3]
C:\Program Files\Splunk\etc\apps\CDN_MKXLVI\local\props.conf TRANSFORMS-nullQmkXLVI = nullFiltermkXLVI

transform.conf:

C:\Program Files\Splunk\etc\apps\CDN_MKXLVI\local\transforms.conf [nullFiltermkXLVI]
C:\Program Files\Splunk\etc\apps\CDN_MKXLVI\local\transforms.conf DEST_KEY = queue
C:\Program Files\Splunk\etc\apps\CDN_MKXLVI\local\transforms.conf FORMAT = nullQueue
C:\Program Files\Splunk\etc\apps\CDN_MKXLVI\local\transforms.conf REGEX = WebServices

This is my 46th attempt with different variations on how I'm referencing sourcetype, regex style and other pieces and I have been unable to get anything to filter out these records.

My input is an s3 bucket and I'm using the splunk addon for AWS (http://docs.splunk.com/Documentation/AddOns/latest/AWS/Description) Can anyone take a look at what I'm up to here and let me know what I can try to sort this out? My assumption is that I'm not wiring something up correctly in my conf files and the regex is never being applied but I've been failing for over a day now so anything you can tell me to try would be grand.

Thanks,
Pat

Re: Routing to nullQueue - version 6.2

sfmike — Wed, 30 Sep 2015 20:51:43 GMT

Can you provide the reference you used to come up with the inputs configuration?

Thanks, Mike

Re: Routing to nullQueue - version 6.2

pattypayscale — Wed, 30 Sep 2015 21:12:41 GMT

This is an S3 input from the aws addon. Here is the link to the s3 portion of the addon: http://docs.splunk.com/Documentation/AddOns/latest/AWS/S3

Re: Routing to nullQueue - version 6.2

the_wolverine — Thu, 01 Oct 2015 14:23:20 GMT

The props.conf syntax for referencing a sourcetype is [mkXLVI:was:s3], not [sourcetype::mkXLVI:was:s3]
Once you update that make sure that the configuration exists either on the indexer (if you are using light/universal forwarder) or on the heavy forwarder. This is parsing that cannot be performed by a universal/light forwarder. If you don't know whether you're using a light or heavy forwarder there are ways to figure that out, but if in doubt just place the same configuration in both locations.

So the following configuration should work:

props.conf

[mkXLVI:aws:s3]
TRANSFORMS-null = nullFiltermkXLVI

transforms.conf:

[nullFiltermkXLVI]
DEST_KEY = queue
FORMAT = nullQueue
REGEX = WebServices

Re: Routing to nullQueue - version 6.2

pattypayscale — Fri, 02 Oct 2015 16:47:44 GMT

Thanks man, I was trying this yesterday and was getting no results at all, which is equally wrong but in a fun and different way. I've got more time to work on this today. I appreciate your help.

Re: Routing to nullQueue - version 6.2

pattypayscale — Fri, 02 Oct 2015 19:59:49 GMT

Right, something else, obvs, was preventing data being picked up. Works now! Thanks a bunch man!