I have the universal forwarder pushed out to some Apache web servers that are indexing some access logs. I would like to send events that represent status checks to nullQueue so they are not indexed. Seems like a pretty simple task to accomplish, but inspection of the logs confirms the events are still being indexed. Here is my props/transforms on my indexers:
Here is a sample event:
10.10.10.10 - - [08/Oct/2018:14:51:33 -0500] "GET /heartbeat_flow HTTP/1.1" 200 7 "-" "-" - -
Here is my props/transforms on my indexers:
[access_combined]
TRANSFORMS-SendHealthChecksToNull = SendHealthChecksToNull1,SendHealthChecksToNull2
[SendHealthChecksToNull1]
REGEX = GET (?:\/.*)?\/(?:DateServlet|dateservlet.ashx|ping)\/? HTTP\/1.1
DEST_KEY = queue
FORMAT = nullQueue
[SendHealthChecksToNull2]
REGEX = GET (?:\/secure\/webmon\/monitor.html|\/heartbeat_flow|\/wps\/portal\/dpath\/monitor|\/webmon\/test.html|\/mf_monitor|\/applicationDBcheck.php|\/check.txt) HTTP\/1.1
DEST_KEY = queue
FORMAT = nullQueue
What am I doing wrong?
can you try first only one like below to check if it is working-
in props.conf
[access_combined]
TRANSFORMS-SendHealthChecksToNull = SendHealthChecksToNull2
in transforms.conf -
[SendHealthChecksToNull2]
REGEX = GET\s\/(secure|webmon|monitor\.html|heartbeat_flow|wps|portal|dpath|monitor|test\.html|mf_monitor|applicationDBcheck\.php|check\.txt)\sHTTP\/1.1
DEST_KEY = queue
FORMAT = nullQueue
Hi scottprigge,
have you an Heavy Forwarder between Universal Forwarder and Indexers?
if yes, you have to put your filter on the Heavy Forwarder.
Bye.
Giuseppe
No, there is no HF in play. It's just UF on the web servers forwarding to the indexers.
Ok,
so try to have two different commands in props.conf:
[access_combined]
TRANSFORMS-SendHealthChecksToNull1 = SendHealthChecksToNull1
TRANSFORMS-SendHealthChecksToNull2 = SendHealthChecksToNull2
Bye.
Giuseppe
can you try first only one like below to check if it is working-
in props.conf
[access_combined]
TRANSFORMS-SendHealthChecksToNull = SendHealthChecksToNull2
in transforms.conf -
[SendHealthChecksToNull2]
REGEX = GET\s\/(secure|webmon|monitor\.html|heartbeat_flow|wps|portal|dpath|monitor|test\.html|mf_monitor|applicationDBcheck\.php|check\.txt)\sHTTP\/1.1
DEST_KEY = queue
FORMAT = nullQueue
The issue was using a space instead of \s in the REGEX stanza. Thanks for the post!
@scottprigge, have you tried this on indexer?
Yes, all the config I referenced is on the indexer.
Have you verified the regex works in a tool like regex101.com?
Yep, I have. It matches.