How come Apache web server logs being sent to nullQueue are still being indexed?

[_smp_]

I have the universal forwarder pushed out to some Apache web servers that are indexing some access logs. I would like to send events that represent status checks to nullQueue so they are not indexed. Seems like a pretty simple task to accomplish, but inspection of the logs confirms the events are still being indexed. Here is my props/transforms on my indexers:

Here is a sample event:
10.10.10.10 - - [08/Oct/2018:14:51:33 -0500] "GET /heartbeat_flow HTTP/1.1" 200 7 "-" "-" - -

Here is my props/transforms on my indexers:
[access_combined]
TRANSFORMS-SendHealthChecksToNull = SendHealthChecksToNull1,SendHealthChecksToNull2

[SendHealthChecksToNull1]
REGEX = GET (?:\/.*)?\/(?:DateServlet|dateservlet.ashx|ping)\/? HTTP\/1.1
DEST_KEY = queue
FORMAT = nullQueue

[SendHealthChecksToNull2]
REGEX = GET (?:\/secure\/webmon\/monitor.html|\/heartbeat_flow|\/wps\/portal\/dpath\/monitor|\/webmon\/test.html|\/mf_monitor|\/applicationDBcheck.php|\/check.txt) HTTP\/1.1
DEST_KEY = queue
FORMAT = nullQueue

What am I doing wrong?

[gcusello]

Hi scottprigge,
have you an Heavy Forwarder between Universal Forwarder and Indexers?
if yes, you have to put your filter on the Heavy Forwarder.
Bye.
Giuseppe

[_smp_]

No, there is no HF in play. It's just UF on the web servers forwarding to the indexers.

[gcusello]

Ok,
so try to have two different commands in props.conf:

[access_combined]
TRANSFORMS-SendHealthChecksToNull1 = SendHealthChecksToNull1
TRANSFORMS-SendHealthChecksToNull2 = SendHealthChecksToNull2

Bye.
Giuseppe

[493669 · Accepted solution]

can you try first only one like below to check if it is working-
in props.conf

[access_combined]
TRANSFORMS-SendHealthChecksToNull = SendHealthChecksToNull2

in transforms.conf -

[SendHealthChecksToNull2]
REGEX = GET\s\/(secure|webmon|monitor\.html|heartbeat_flow|wps|portal|dpath|monitor|test\.html|mf_monitor|applicationDBcheck\.php|check\.txt)\sHTTP\/1.1
DEST_KEY = queue
FORMAT = nullQueue

[_smp_]

The issue was using a space instead of \s in the REGEX stanza. Thanks for the post!

[493669]

@scottprigge, have you tried this on indexer?

[_smp_]

Yes, all the config I referenced is on the indexer.

[kmorris_splunk]

Have you verified the regex works in a tool like regex101.com?

[_smp_]

Yep, I have. It matches.