Filtering out 90% log events from indexing

[strive]

Hi,

We are using Splunk 6.1.1 in one of our products. In two of our other products where we use splunk 6.0 and 5.4, we have filtered log events from indexing using configurations in props.conf and transforms.conf. Here the percentage of log events sent to nullQueue is less (say 10%).

In the new product we are estimated to get 400 million log events per day and we are interested in only 40 million log events. We do not want to index remaining 360 million log events. Similar to our other products we are planning to filter these events by appropriate configurations in props.conf and transforms.conf. Is this the right approach to filter events? OR is there any better way to filter syslog events.

Thanks

Strive

[dshpritz · Accepted solution]

The best way to filter is to not ingest the items to start with. So, if you are feeding the syslog feed into a syslog receiver (such as syslog-ng or rsyslogd) you can use that to break the syslog feed into different files, then only ingest the files you are interested in. If you are not able to do that, then props and transforms are your best bet.

[strive]

Unfortunately we cannot break the syslog feed.