We have a clustered environment that includes heavy forwarders, universal forwarders, and forwarders under Windows. The development team sometimes do performance tests and these generate a lot of data that we don't want to be indexed. We could add a new rule on the heavy forwarders to send to null queue all events during the tests , but can this be done at forwarder or universal forwarder level? Do you think that there is a better way to achieve this ?
You can have the data indexed into specific indexes or add a specific field which indicates that this is a performance test data. Then it's easy to "simply" delete this type of data.
Have a look at this Splunk documentation to know more about event routing and filter.
The send to null queue can be done on universal forwarder if it's to be done without looking into individual events (purely based on index/source/sourcetype/host). If you need to look at the event data to filter, than you need to do routing/filtering in heavy forwarder/indexer
It looks like this only can be done at hf or indexer level as I suspected, but not in universal forwarder:
"Although similar to forwarder-based routing, queue routing can be performed by an indexer, as well as a heavy forwarder"