Whitelist specific pod in namespace for Heavy Forwarder

[jarelloy]

I have a namespace, hammy, in k8s with several pods deployed. Out of all the pods, i'd only want the pods with the name "blausy" to be ingested into the Heavy Forwarder. The rest of the logs in other pods should be dropped. How should i do it? Thanks!

The log's source is something like

source = /var/log/pods/hammy_blausy-api-<uuid>

Inside my props.conf, this is my configuration

[source::/var/log/pods/hammy_*]
TRANSFORMS-routing = whitelist_blausy_logs, drop_all_logs

transform.conf

[whitelist_blausy_logs]
SOURCE_KEY = MetaData:Source
REGEX = blausy-*
DEST_KEY = queue
FORMAT = indexQueue

[drop_all_logs]
REGEX = .*
DEST_KEY = queue
FORMAT = nullQueue

Edit: at the moment, it's either dropping all the logs in hammy namespace, or ingesting all the logs

[thahir]

@jarelloy As @PickleRick  said, in simple terms drop everything first and then pull back what you want exactly.

try the below 

props.conf

[source::/var/log/pods/hammy_*]
TRANSFORMS-routing = set_null_for_hammy, whitelist_blausy

transforms.conf

[set_null_for_hammy]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[whitelist_blausy]
SOURCE_KEY = MetaData:Source
REGEX = /var/log/pods/hammy_blausy
DEST_KEY = queue
FORMAT = indexQueue

[PickleRick]

Just for the sake of clarity (and that's probably one of the reason there is so much confusion about this).

It's not that you "drop" or "pull" the events. If you dropped the events you wouldn't have anything to pull back.

Transforms just set metadata field which is used _after_ to route events. So you can manipulate this metadata at will and switch it back and forth. After you leave the transforms phase the data is getting routed or dropped according to the final state of the metadata fields.

[PickleRick · Accepted solution]

This is a very typical mistake (don't get it personally, everyone makes it at some point ;-)).

A list of transforms - in your case it's TRANSFORMS-routing - contains a list of transforms of which all will be executed. It's not something like ACL on a router when you terminate the list when you match some condition but it is always evaluated from left to right, from beginning to the end.

I assume the misconception comes from the fact that transforms themselves do not actually drop the events but rather set metadata fields on which Splunk acts later after the event passes this stage of the pipeline. 

So in your case you firstly send some events to indexQueue (which is a bit redundant since they are destined there by default) and then you send all events to dropQueue (including those you previously "redirected" to indexQueue. And only after the transforms have been finished Splunk finally does something with the event according to the metadata fields. In your case, since all events have destination queue specified as dropQueue, all events are dropped.

Do you see where I'm going with this?

The order should be reversed. First you should "drop" all events (set the destination queue to dropQueue). And then, on events which are supposed to be retained, you overwrite that value with indexQueue.

[source::/var/log/pods/hammy_*]
TRANSFORMS-routing = drop_all_logs, whitelist_blausy_logs

[jarelloy]

thanks so much for help!