Getting Data In

Routing to nullQueue - version 6.2

Explorer

Hello All,
I am attempting to filter out specific events from a given input, they're useless and I don't want to waste resources indexing them. I have found a number of seemingly great answers and help pages for how to set this up with a props.conf and transforms.conf. Below are the dumps from btool for my app's inputs.conf, props.conf and transforms.conf:

inputs.conf:

C:\Program Files\Splunk\etc\apps\CDNMKXLVI\local\inputs.conf [awss3://cdn-mkXLVI]
C:\Program Files\Splunk\etc\apps\CDNMKXLVI\local\inputs.conf awsaccount = SplunkIAM
C:\Program Files\Splunk\etc\apps\CDNMKXLVI\local\inputs.conf bucketname = testbucket
C:\Program Files\Splunk\etc\apps\CDNMKXLVI\local\inputs.conf keyname = cflogs
C:\Program Files\Splunk\etc\apps\CDN_MKXLVI\local\inputs.conf sourcetype = mkXLVI:aws:s3

props.conf:

C:\Program Files\Splunk\etc\apps\CDNMKXLVI\local\props.conf [sourcetype::mkXLVI:aws:s3]
C:\Program Files\Splunk\etc\apps\CDN
MKXLVI\local\props.conf TRANSFORMS-nullQmkXLVI = nullFiltermkXLVI

transform.conf:

C:\Program Files\Splunk\etc\apps\CDNMKXLVI\local\transforms.conf [nullFiltermkXLVI]
C:\Program Files\Splunk\etc\apps\CDN
MKXLVI\local\transforms.conf DESTKEY = queue
C:\Program Files\Splunk\etc\apps\CDN
MKXLVI\local\transforms.conf FORMAT = nullQueue
C:\Program Files\Splunk\etc\apps\CDN_MKXLVI\local\transforms.conf REGEX = WebServices

This is my 46th attempt with different variations on how I'm referencing sourcetype, regex style and other pieces and I have been unable to get anything to filter out these records.

My input is an s3 bucket and I'm using the splunk addon for AWS (http://docs.splunk.com/Documentation/AddOns/latest/AWS/Description) Can anyone take a look at what I'm up to here and let me know what I can try to sort this out? My assumption is that I'm not wiring something up correctly in my conf files and the regex is never being applied but I've been failing for over a day now so anything you can tell me to try would be grand.

Thanks,
Pat

1 Solution

Champion

The props.conf syntax for referencing a sourcetype is [mkXLVI:was:s3], not [sourcetype::mkXLVI:was:s3]
Once you update that make sure that the configuration exists either on the indexer (if you are using light/universal forwarder) or on the heavy forwarder. This is parsing that cannot be performed by a universal/light forwarder. If you don't know whether you're using a light or heavy forwarder there are ways to figure that out, but if in doubt just place the same configuration in both locations.

So the following configuration should work:

props.conf

[mkXLVI:aws:s3]
TRANSFORMS-null = nullFiltermkXLVI

transforms.conf:

[nullFiltermkXLVI]
DEST_KEY = queue
FORMAT = nullQueue
REGEX = WebServices

View solution in original post

Champion

The props.conf syntax for referencing a sourcetype is [mkXLVI:was:s3], not [sourcetype::mkXLVI:was:s3]
Once you update that make sure that the configuration exists either on the indexer (if you are using light/universal forwarder) or on the heavy forwarder. This is parsing that cannot be performed by a universal/light forwarder. If you don't know whether you're using a light or heavy forwarder there are ways to figure that out, but if in doubt just place the same configuration in both locations.

So the following configuration should work:

props.conf

[mkXLVI:aws:s3]
TRANSFORMS-null = nullFiltermkXLVI

transforms.conf:

[nullFiltermkXLVI]
DEST_KEY = queue
FORMAT = nullQueue
REGEX = WebServices

View solution in original post

Explorer

Thanks man, I was trying this yesterday and was getting no results at all, which is equally wrong but in a fun and different way. I've got more time to work on this today. I appreciate your help.

0 Karma

Explorer

Right, something else, obvs, was preventing data being picked up. Works now! Thanks a bunch man!

0 Karma

Explorer

Can you provide the reference you used to come up with the inputs configuration?

Thanks, Mike

0 Karma

Explorer

This is an S3 input from the aws addon. Here is the link to the s3 portion of the addon: http://docs.splunk.com/Documentation/AddOns/latest/AWS/S3

0 Karma