Getting Data In

How do you exclude log lines from being indexed?

krisbent
New Member

Hi, I am using Splunk 6.5.
How can I exclude lines containing a pattern from being indexed? In my case I have IIS access logs forwarded by a Universal Forwarder. I have tried to configure like this, but log lines that contains bigip is still indexed.

system/default/props.conf
[iis]
INDEXED_EXTRACTIONS = w3c

system/local/props.conf
[iis]
TRANSFORMS-null=ignorebigip

system/local/transforms.conf
[ignorebigip]
REGEX = (?m)^.(bigip)\s.$
DEST_KEY = queue
FORMAT = nullQueue

If I understand this answer https://answers.splunk.com/answers/453417/parse-iis-logs-structured-data-on-universal-forwar.html , it is not possible to send to the nullQueue when the "standard" [iis] sourcetype with INDEXED_EXTRACTIONS = w3c.

Is that true, do I really have to configure how to extract the fields the "pre-Splunk 6"-way?

0 Karma

kmorris_splunk
Splunk Employee
Splunk Employee

It isn't possible to filter data using a Universal Forwarder, with the exception of Blacklisting or Whitelisting Wiindows event codes. You would need to use the props and transforms settings on the indexers, or use a Heavy Forwarder.

If using a Heavy Forwarder, you need to consider how much of the data you are actually filtering out. If it isn't a large percentage, then it isn't worth it since Heavy Forwarders send what is called "cooked data" which is larger than what a Universal Forwarder would send. So you really wouldn't be cutting back on any network traffic.

If you aren't filtering a large portion, use the Universal Forwarder and add the props.conf and transforms.conf settings to your indexers.

Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...