How do you exclude log lines from being indexed?

krisbent — Thu, 24 Aug 2017 10:37:21 GMT

Hi, I am using Splunk 6.5.
How can I exclude lines containing a pattern from being indexed? In my case I have IIS access logs forwarded by a Universal Forwarder. I have tried to configure like this, but log lines that contains bigip is still indexed.

system/default/props.conf
[iis]
INDEXED_EXTRACTIONS = w3c

system/local/props.conf
[iis]
TRANSFORMS-null=ignorebigip

system/local/transforms.conf
[ignorebigip]
REGEX = (?m)^.(bigip)\s.$
DEST_KEY = queue
FORMAT = nullQueue

If I understand this answer https://answers.splunk.com/answers/453417/parse-iis-logs-structured-data-on-universal-forwar.html , it is not possible to send to the nullQueue when the "standard" [iis] sourcetype with INDEXED_EXTRACTIONS = w3c.

Is that true, do I really have to configure how to extract the fields the "pre-Splunk 6"-way?

Re: How do you exclude log lines from being indexed?

kmorris_splunk — Fri, 25 Aug 2017 13:53:04 GMT

It isn't possible to filter data using a Universal Forwarder, with the exception of Blacklisting or Whitelisting Wiindows event codes. You would need to use the props and transforms settings on the indexers, or use a Heavy Forwarder.

If using a Heavy Forwarder, you need to consider how much of the data you are actually filtering out. If it isn't a large percentage, then it isn't worth it since Heavy Forwarders send what is called "cooked data" which is larger than what a Universal Forwarder would send. So you really wouldn't be cutting back on any network traffic.

If you aren't filtering a large portion, use the Universal Forwarder and add the props.conf and transforms.conf settings to your indexers.

topic Re: How do you exclude log lines from being indexed? in Getting Data In

How do you exclude log lines from being indexed?

Re: How do you exclude log lines from being indexed?