Getting Data In

dropping events at index time works from one forwarder but not from the other

gesa_behrens
Path Finder

My scenario is:
1 Indexer (SPLUNK Enterprise 7.1.3)
1 Heavy Forwarder (SPLUNK Enterprise 7.1.2
1 Universal Forwarder (7.1.2)

both forwarders forward /var/log/tomcat7/catalina.out to the same index with the same sourcetype at the indexer instance.
Both files have the same structure containing 3 different types of events :

 [DEBUG] [15.10.2018] [10:40:04] [CustomerController] ...
 2018-10-15 10:42:37,875 [ajp-bio-8009-exec-10] INFO...
 Oct 15, 2018 8:05:14 AM org.apache.catalina.startup.HostConfig...

I want to index only the firs type. So on indexer side I made the following configuration entries :
in props.conf :

 [source::/var/log/tomcat7/catalina.out]
 TRANSFORMS-set=setnull,setparsing

in transforms.conf :

 [setnull]
 REGEX = .
 DEST_KEY = queue
 FORMAT = nullQueue

 [setparsing]
 REGES = ^\[[A_Z]*\]
 DEST_KEY = queue
 FORMAT = indexQueue

The data from the Universal Forwarder is filtered as expected, only the first type events are indexed
The data from the Heavy Forwarder is not filtered, all events are indexed.

I restarted all the instances several times.
I compared the outputs.conf from both forwarders, they are identical.

Any ideas what the reason for the different behavior could be?

Thanks!

gesa_behrens
Path Finder

Hi harsmarvania57,

No, both forwarders send directly to indexer.
The reason for one being a heavy forwarder is that on this machine we also need to collect data from a data base.

But this still is a good idea, I will try to drop the events at heavy forwarder side, so they won't be sent
I'll you know if it worked.
Thanks!

0 Karma

harsmarvania57
Ultra Champion

Hi @gesa_behrens,

How UF is sending data to Splunk Indexer ? Is UF sending logs to Heavy Forwarder and then Heavy Forwarder sends data to Indexer ? If this is the case then props.conf and transforms.conf configuration should be on Heavy Forwarder, not on Indexer because parsing happens on first Splunk Enterprise Instance from Universal Forwarder.

EDIT: Also if you want to match line with [DEBUG] then in transforms.conf, Regular Expression should be like this REGES = ^\[[A-Z]*\]in [setparsing] stanza.

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...