troubleshooting filtering at Heavy Forwarder with Props.conf / Transform.conf

[k2skaterii]

I am currently passing all logs through a Heavy Forwarder so I can filter out "noisy" logs before they are indexed. I am successfully filtering 4 other items including Windows Logs, SYSLOG, and Windows WMI data. Now I am trying to filter IIS logs, and am running into trouble.

I believe my props.conf and transforms.conf are setup correctly. Can someone help me figure out where this filter is failing?

Props.conf:
[iis]
TRANSFORMS-null2=IIS_Remove_F5_Health_Checks

---

transforms.conf:
[IIS_Remove_F5_Health_Checks]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueu

---

I realize my REGEX is going to filter EVERYTHING. I had a more specific REGEX in there previously, but I replaced it with the " REGEX = ." as part of my troubleshooting process. The original REGEX was supposed to filter out any logs with a 10.1.100.8 or 10.1.100.9 IP address, that REGEX was:

REGEX=(?<!\d)10.(?<!\d)1.(?<!\d)100.(?<!\d)(8|9)

The sourcetype for my ISS logs is "iis" (all lower case). In my mind it looks like all iis logs should be passing processed by the transform, but it looks like that is not the case. How can I troubleshoot this?

Thanks for the help!

[dwaddle]

Are your IIS logs at the forwarder using INDEXED_EXTRACTIONS? If so, data arrives at the heavy "cooked" already. Configure the forwarder to nullQueue it. Seems wrong, I know. But..

[k2skaterii]

The host that is forwarding the iis logs is using the default props.conf, which includes:

INDEXED_EXTRACTIONS = w3c

I do want INDEXED_EXTRACTIONS TO be active as that helps the logs be "Useful". If I disabled INDEXED_EXTRACTIONS on each of my host would the Heavy Forwarder be able to provide that function?

[dshpritz]

Yes, if you are not using indexed extractions, then the data is not sent cooked, so the HF will be able to filter.

[dwaddle]

Or, configure your nullQueue entries for [iis] directly on the UFs themselves. They will filter BEFORE things get to the heavy.