topic Filtering during index-time not working in Getting Data In

Filtering during index-time not working

spock_yh — Mon, 21 Mar 2011 09:36:22 GMT

I have the following stanza in transforms.conf:

[medusa_media_access-drop-events]
REGEX = ^\S+\s++\S+\s++\[[^\]]*\]\s++\S+\s++"[^"]*"\s++POST\s++"/soap"\s++""\s++200\s++
DEST_KEY = queue
FORMAT = nullQueue

And the following in props.conf:

[medusa_media_access]
REPORT-extractions = medusa_media_access-extractions,medusa_media_access-uri-fields,medusa_media_access-query-fields,medusa_media_access-legacy_medialookup-primary_key
TRANSFORMS-drop = medusa_media_access-drop-events
FIELDALIAS-legacy = accountId AS medusa_account_alias
KV_MODE=none

The setting for index-time filtering doesn't seem to filter any events, even those the pcregextest matches. For example:

1.1.1.1 - [20/Mar/2011:19:18:06 -0700] 15625 "my.site.com" POST "/soap" "" 200 590 "medusa" "-" "Apache CXF 2.2.9" "2.2.2.2"

The other settings for the sourcetype (the search-time extractions, for example), do apply. I have this configured on a single indexer + search head, with the log data itself being sent from two remote forwarders.

Any idea on what I'm missing?

Re: Filtering during index-time not working

Ayn — Mon, 21 Mar 2011 12:54:37 GMT

Your props.conf / transforms.conf setup looks fine to me. However I just checked your regex and it seems you have lots of superfluous '+' characters in it - are those in the original regex as well or was it some kind of typo when you entered text here on the site?

Also, are the remote forwarders light or heavy forwarders? If they're heavy forwarders you'll want to perform this filtering on the forwarders themselves instead.

Re: Filtering during index-time not working

spock_yh — Fri, 25 Mar 2011 01:55:15 GMT

Hi Ayn

The forwarders are universal forwarders (4.2).

Regarding the superfluous '+' characters, the regex was basically copied from the default regex for "access-extractions" in the default props.conf. The double '+' means that the regex shouldn't backtrack after matching the spaces.

Anyway, I find it hard to believe that the problem is the regex because pcregextest does claim it is a match.