Getting Data In

How come Apache web server logs being sent to nullQueue are still being indexed?

_smp_
Builder

I have the universal forwarder pushed out to some Apache web servers that are indexing some access logs. I would like to send events that represent status checks to nullQueue so they are not indexed. Seems like a pretty simple task to accomplish, but inspection of the logs confirms the events are still being indexed. Here is my props/transforms on my indexers:

Here is a sample event:
10.10.10.10 - - [08/Oct/2018:14:51:33 -0500] "GET /heartbeat_flow HTTP/1.1" 200 7 "-" "-" - -

Here is my props/transforms on my indexers:
[access_combined]
TRANSFORMS-SendHealthChecksToNull = SendHealthChecksToNull1,SendHealthChecksToNull2

[SendHealthChecksToNull1]
REGEX = GET (?:\/.*)?\/(?:DateServlet|dateservlet.ashx|ping)\/? HTTP\/1.1
DEST_KEY = queue
FORMAT = nullQueue

[SendHealthChecksToNull2]
REGEX = GET (?:\/secure\/webmon\/monitor.html|\/heartbeat_flow|\/wps\/portal\/dpath\/monitor|\/webmon\/test.html|\/mf_monitor|\/applicationDBcheck.php|\/check.txt) HTTP\/1.1
DEST_KEY = queue
FORMAT = nullQueue

What am I doing wrong?

0 Karma
1 Solution

493669
Super Champion

can you try first only one like below to check if it is working-
in props.conf

[access_combined]
TRANSFORMS-SendHealthChecksToNull = SendHealthChecksToNull2

in transforms.conf -

[SendHealthChecksToNull2]
REGEX = GET\s\/(secure|webmon|monitor\.html|heartbeat_flow|wps|portal|dpath|monitor|test\.html|mf_monitor|applicationDBcheck\.php|check\.txt)\sHTTP\/1.1
DEST_KEY = queue
FORMAT = nullQueue

View solution in original post

gcusello
SplunkTrust
SplunkTrust

Hi scottprigge,
have you an Heavy Forwarder between Universal Forwarder and Indexers?
if yes, you have to put your filter on the Heavy Forwarder.
Bye.
Giuseppe

0 Karma

_smp_
Builder

No, there is no HF in play. It's just UF on the web servers forwarding to the indexers.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Ok,
so try to have two different commands in props.conf:

[access_combined]
TRANSFORMS-SendHealthChecksToNull1 = SendHealthChecksToNull1
TRANSFORMS-SendHealthChecksToNull2 = SendHealthChecksToNull2

Bye.
Giuseppe

0 Karma

493669
Super Champion

can you try first only one like below to check if it is working-
in props.conf

[access_combined]
TRANSFORMS-SendHealthChecksToNull = SendHealthChecksToNull2

in transforms.conf -

[SendHealthChecksToNull2]
REGEX = GET\s\/(secure|webmon|monitor\.html|heartbeat_flow|wps|portal|dpath|monitor|test\.html|mf_monitor|applicationDBcheck\.php|check\.txt)\sHTTP\/1.1
DEST_KEY = queue
FORMAT = nullQueue

_smp_
Builder

The issue was using a space instead of \s in the REGEX stanza. Thanks for the post!

493669
Super Champion

@scottprigge, have you tried this on indexer?

0 Karma

_smp_
Builder

Yes, all the config I referenced is on the indexer.

0 Karma

kmorris_splunk
Splunk Employee
Splunk Employee

Have you verified the regex works in a tool like regex101.com?

0 Karma

_smp_
Builder

Yep, I have. It matches.

0 Karma
Get Updates on the Splunk Community!

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...