Getting Data In

Routing to nullQueue - version 6.2

pattypayscale
Explorer

Hello All,
I am attempting to filter out specific events from a given input, they're useless and I don't want to waste resources indexing them. I have found a number of seemingly great answers and help pages for how to set this up with a props.conf and transforms.conf. Below are the dumps from btool for my app's inputs.conf, props.conf and transforms.conf:

inputs.conf:

C:\Program Files\Splunk\etc\apps\CDN_MKXLVI\local\inputs.conf [aws_s3://cdn-mkXLVI]
C:\Program Files\Splunk\etc\apps\CDN_MKXLVI\local\inputs.conf aws_account = SplunkIAM
C:\Program Files\Splunk\etc\apps\CDN_MKXLVI\local\inputs.conf bucket_name = testbucket
C:\Program Files\Splunk\etc\apps\CDN_MKXLVI\local\inputs.conf key_name = cflogs
C:\Program Files\Splunk\etc\apps\CDN_MKXLVI\local\inputs.conf sourcetype = mkXLVI:aws:s3

props.conf:

C:\Program Files\Splunk\etc\apps\CDN_MKXLVI\local\props.conf [sourcetype::mkXLVI:aws:s3]
C:\Program Files\Splunk\etc\apps\CDN_MKXLVI\local\props.conf TRANSFORMS-nullQmkXLVI = nullFiltermkXLVI

transform.conf:

C:\Program Files\Splunk\etc\apps\CDN_MKXLVI\local\transforms.conf [nullFiltermkXLVI]
C:\Program Files\Splunk\etc\apps\CDN_MKXLVI\local\transforms.conf DEST_KEY = queue
C:\Program Files\Splunk\etc\apps\CDN_MKXLVI\local\transforms.conf FORMAT = nullQueue
C:\Program Files\Splunk\etc\apps\CDN_MKXLVI\local\transforms.conf REGEX = WebServices

This is my 46th attempt with different variations on how I'm referencing sourcetype, regex style and other pieces and I have been unable to get anything to filter out these records.

My input is an s3 bucket and I'm using the splunk addon for AWS (http://docs.splunk.com/Documentation/AddOns/latest/AWS/Description) Can anyone take a look at what I'm up to here and let me know what I can try to sort this out? My assumption is that I'm not wiring something up correctly in my conf files and the regex is never being applied but I've been failing for over a day now so anything you can tell me to try would be grand.

Thanks,
Pat

1 Solution

the_wolverine
Champion

The props.conf syntax for referencing a sourcetype is [mkXLVI:was:s3], not [sourcetype::mkXLVI:was:s3]
Once you update that make sure that the configuration exists either on the indexer (if you are using light/universal forwarder) or on the heavy forwarder. This is parsing that cannot be performed by a universal/light forwarder. If you don't know whether you're using a light or heavy forwarder there are ways to figure that out, but if in doubt just place the same configuration in both locations.

So the following configuration should work:

props.conf

[mkXLVI:aws:s3]
TRANSFORMS-null = nullFiltermkXLVI

transforms.conf:

[nullFiltermkXLVI]
DEST_KEY = queue
FORMAT = nullQueue
REGEX = WebServices

View solution in original post

the_wolverine
Champion

The props.conf syntax for referencing a sourcetype is [mkXLVI:was:s3], not [sourcetype::mkXLVI:was:s3]
Once you update that make sure that the configuration exists either on the indexer (if you are using light/universal forwarder) or on the heavy forwarder. This is parsing that cannot be performed by a universal/light forwarder. If you don't know whether you're using a light or heavy forwarder there are ways to figure that out, but if in doubt just place the same configuration in both locations.

So the following configuration should work:

props.conf

[mkXLVI:aws:s3]
TRANSFORMS-null = nullFiltermkXLVI

transforms.conf:

[nullFiltermkXLVI]
DEST_KEY = queue
FORMAT = nullQueue
REGEX = WebServices

pattypayscale
Explorer

Thanks man, I was trying this yesterday and was getting no results at all, which is equally wrong but in a fun and different way. I've got more time to work on this today. I appreciate your help.

0 Karma

pattypayscale
Explorer

Right, something else, obvs, was preventing data being picked up. Works now! Thanks a bunch man!

0 Karma

sfmike
Explorer

Can you provide the reference you used to come up with the inputs configuration?

Thanks, Mike

0 Karma

pattypayscale
Explorer

This is an S3 input from the aws addon. Here is the link to the s3 portion of the addon: http://docs.splunk.com/Documentation/AddOns/latest/AWS/S3

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

Industry Solutions for Supply Chain and OT, Amazon Use Cases, Plus More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...