Issue with Dropping Events via props.conf and transforms.conf

[Cybers1]

Hi Splunk Community,

We’re currently trying to drop specific logs using props.conf and transforms.conf, but our configuration doesn’t seem to be working as expected. Below is a summary of what we’ve done:

transforms.conf

[eliminate-accesslog_coll_health]
REGEX = ^.*(?:H|h)ealth.*
DEST_KEY = queue
FORMAT = nullQueue

[eliminate-accesslog_coll_actuator]
REGEX = ^.*actuator.*
DEST_KEY = queue
FORMAT = nullQueue

props.conf

[access_combined]
TRANSFORMS-set = eliminate-accesslog_coll_actuator, eliminate-accesslog_coll_health

[iis]
TRANSFORMS-set = eliminate-accesslog_coll_health

[(?::){0}kube:*]
TRANSFORMS-set = eliminate-accesslog_coll_actuator

The main issue is that events are not being dropped, even when a specific sourcetype is defined (like access_combined or iis). Additionally, for logs coming from Kubernetes, there is no single consistent sourcetype, so we attempted to match using [source::] logic via a regex ([(?::){0}kube:*]), but this doesn’t seem to be supported in this context.

From what we've read in the documentation, it looks like regex patterns for [source::] are not allowed in props.conf, and must instead be written explicitly. Is that correct? And if so, what’s the best way to drop events from dynamic sources or where the sourcetype is inconsistent?

Any help or suggestions would be greatly appreciated.

Thanks in advance!

[Cybers1]

Hi, @gcusello 

Thanks for your reply!

The configurations were applied on the main Splunk instance, not on the Heavy Forwarder. However, within the same configuration files on the main instance (props.conf and transforms.conf), there are already similar settings that are currently working as expected.

Also, we have tested the regex patterns used in the transforms.conf, and they are working correctly — they match the intended log lines when tested separately.

That’s why we’re a bit puzzled — some TRANSFORMS stanzas are effective, while the ones mentioned in the original post aren’t having any impact.

Any further insights would be greatly appreciated!

Best regards,

[isoutamo]

Are you sure that e.g. some of your source nodes are not HF instead of UF? This is valid also form collecting node not only any nodes between UF and Indexers!

Basically your configuration seems to be ok. Of course you could modify those REGEX little bit efficient than those are currently, but it's another story.

As other already said the most obvious reason is that you have HF somewhere before your indexer (splunk enterprise node).

You can check this on source node and all other nodes (check those from outputs.conf)

$SPLUNK_HOME/bin/splunk version

 This should show in UF

Splunk Universal Forwarder 9.4.0 (build 6b4ebe426ca6)

or in HF/indexer 

Splunk 9.4.1 (build e3bdab203ac8)

Just check which path it is running and replace SPLUNK_HOME with it. 

[livehybrid]

Hi @Cybers1 

So does this data go through the HF? If so these props/transforms will need putting on there.

Certain props/transforms happen at search time (e.g. search time field extractions), the other config which happens at parsing/index time needs to occur on the first full instance of Splunk that the data hits (e.g. a HF), apart from a vary small set of exceptions (such as RULESET)

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[gcusello]

Hi @Cybers1 ,

if you're sure that the not filtered data don't pass trhough any HF, it isn't the issue.

then, please try these regexes:

[eliminate-accesslog_coll_health]
REGEX = Health|health
DEST_KEY = queue
FORMAT = nullQueue

[eliminate-accesslog_coll_actuator]
REGEX = actuator
DEST_KEY = queue
FORMAT = nullQueue

Ciao.

Giuseppe

[gcusello]

Hi @Cybers1 ,

I suppose that you already checked the regexes that you're using and they are correct to filter logs.

The question is: where do you located these conf files?

They must be located on the first full Splunk instance that dara pass trhough, in other words on the first Heavy Forwarder (if present) or in the Indexers.

Ciao.

Giuseppe