Getting Data In

How come Apache web server logs being sent to nullQueue are still being indexed?

_smp_
Builder

I have the universal forwarder pushed out to some Apache web servers that are indexing some access logs. I would like to send events that represent status checks to nullQueue so they are not indexed. Seems like a pretty simple task to accomplish, but inspection of the logs confirms the events are still being indexed. Here is my props/transforms on my indexers:

Here is a sample event:
10.10.10.10 - - [08/Oct/2018:14:51:33 -0500] "GET /heartbeat_flow HTTP/1.1" 200 7 "-" "-" - -

Here is my props/transforms on my indexers:
[access_combined]
TRANSFORMS-SendHealthChecksToNull = SendHealthChecksToNull1,SendHealthChecksToNull2

[SendHealthChecksToNull1]
REGEX = GET (?:\/.*)?\/(?:DateServlet|dateservlet.ashx|ping)\/? HTTP\/1.1
DEST_KEY = queue
FORMAT = nullQueue

[SendHealthChecksToNull2]
REGEX = GET (?:\/secure\/webmon\/monitor.html|\/heartbeat_flow|\/wps\/portal\/dpath\/monitor|\/webmon\/test.html|\/mf_monitor|\/applicationDBcheck.php|\/check.txt) HTTP\/1.1
DEST_KEY = queue
FORMAT = nullQueue

What am I doing wrong?

0 Karma
1 Solution

493669
Super Champion

can you try first only one like below to check if it is working-
in props.conf

[access_combined]
TRANSFORMS-SendHealthChecksToNull = SendHealthChecksToNull2

in transforms.conf -

[SendHealthChecksToNull2]
REGEX = GET\s\/(secure|webmon|monitor\.html|heartbeat_flow|wps|portal|dpath|monitor|test\.html|mf_monitor|applicationDBcheck\.php|check\.txt)\sHTTP\/1.1
DEST_KEY = queue
FORMAT = nullQueue

View solution in original post

gcusello
SplunkTrust
SplunkTrust

Hi scottprigge,
have you an Heavy Forwarder between Universal Forwarder and Indexers?
if yes, you have to put your filter on the Heavy Forwarder.
Bye.
Giuseppe

0 Karma

_smp_
Builder

No, there is no HF in play. It's just UF on the web servers forwarding to the indexers.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Ok,
so try to have two different commands in props.conf:

[access_combined]
TRANSFORMS-SendHealthChecksToNull1 = SendHealthChecksToNull1
TRANSFORMS-SendHealthChecksToNull2 = SendHealthChecksToNull2

Bye.
Giuseppe

0 Karma

493669
Super Champion

can you try first only one like below to check if it is working-
in props.conf

[access_combined]
TRANSFORMS-SendHealthChecksToNull = SendHealthChecksToNull2

in transforms.conf -

[SendHealthChecksToNull2]
REGEX = GET\s\/(secure|webmon|monitor\.html|heartbeat_flow|wps|portal|dpath|monitor|test\.html|mf_monitor|applicationDBcheck\.php|check\.txt)\sHTTP\/1.1
DEST_KEY = queue
FORMAT = nullQueue

_smp_
Builder

The issue was using a space instead of \s in the REGEX stanza. Thanks for the post!

493669
Super Champion

@scottprigge, have you tried this on indexer?

0 Karma

_smp_
Builder

Yes, all the config I referenced is on the indexer.

0 Karma

kmorris_splunk
Splunk Employee
Splunk Employee

Have you verified the regex works in a tool like regex101.com?

0 Karma

_smp_
Builder

Yep, I have. It matches.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...