topic Re: How come Apache web server logs being sent to nullQueue are still being indexed? in Getting Data In

How come Apache web server logs being sent to nullQueue are still being indexed?

_smp_ — Tue, 29 Sep 2020 21:34:30 GMT

I have the universal forwarder pushed out to some Apache web servers that are indexing some access logs. I would like to send events that represent status checks to nullQueue so they are not indexed. Seems like a pretty simple task to accomplish, but inspection of the logs confirms the events are still being indexed. Here is my props/transforms on my indexers:

Here is a sample event:
10.10.10.10 - - [08/Oct/2018:14:51:33 -0500] "GET /heartbeat_flow HTTP/1.1" 200 7 "-" "-" - -

Here is my props/transforms on my indexers:
[access_combined]
TRANSFORMS-SendHealthChecksToNull = SendHealthChecksToNull1,SendHealthChecksToNull2

[SendHealthChecksToNull1]
REGEX = GET (?:\/.*)?\/(?:DateServlet|dateservlet.ashx|ping)\/? HTTP\/1.1
DEST_KEY = queue
FORMAT = nullQueue

What am I doing wrong?

Re: How come Apache web server logs being sent to nullQueue are still being indexed?

kmorris_splunk — Mon, 08 Oct 2018 20:43:04 GMT

Have you verified the regex works in a tool like regex101.com?

Re: How come Apache web server logs being sent to nullQueue are still being indexed?

_smp_ — Mon, 08 Oct 2018 20:51:17 GMT

Yep, I have. It matches.

Re: How come Apache web server logs being sent to nullQueue are still being indexed?

493669 — Tue, 09 Oct 2018 04:29:45 GMT

can you try first only one like below to check if it is working-
in props.conf

[access_combined]
TRANSFORMS-SendHealthChecksToNull = SendHealthChecksToNull2

in transforms.conf -

[SendHealthChecksToNull2]
REGEX = GET\s\/(secure|webmon|monitor\.html|heartbeat_flow|wps|portal|dpath|monitor|test\.html|mf_monitor|applicationDBcheck\.php|check\.txt)\sHTTP\/1.1
DEST_KEY = queue
FORMAT = nullQueue

Re: How come Apache web server logs being sent to nullQueue are still being indexed?

gcusello — Tue, 09 Oct 2018 07:49:21 GMT

Hi scottprigge,
have you an Heavy Forwarder between Universal Forwarder and Indexers?
if yes, you have to put your filter on the Heavy Forwarder.
Bye.
Giuseppe

Re: How come Apache web server logs being sent to nullQueue are still being indexed?

_smp_ — Tue, 09 Oct 2018 12:21:59 GMT

No, there is no HF in play. It's just UF on the web servers forwarding to the indexers.

Re: How come Apache web server logs being sent to nullQueue are still being indexed?

493669 — Tue, 09 Oct 2018 12:33:03 GMT

@scottprigge, have you tried this on indexer?

Re: How come Apache web server logs being sent to nullQueue are still being indexed?

gcusello — Tue, 09 Oct 2018 12:35:57 GMT

Ok,
so try to have two different commands in props.conf:

[access_combined]
TRANSFORMS-SendHealthChecksToNull1 = SendHealthChecksToNull1
TRANSFORMS-SendHealthChecksToNull2 = SendHealthChecksToNull2

Bye.
Giuseppe

Re: How come Apache web server logs being sent to nullQueue are still being indexed?

_smp_ — Tue, 09 Oct 2018 12:39:57 GMT

Yes, all the config I referenced is on the indexer.

Re: How come Apache web server logs being sent to nullQueue are still being indexed?

_smp_ — Tue, 09 Oct 2018 12:41:24 GMT

The issue was using a space instead of \s in the REGEX stanza. Thanks for the post!