topic What the best strategy to discard all temporary data while testing on some forwarders? in Getting Data In

What the best strategy to discard all temporary data while testing on some forwarders?

dsmc_adv — Tue, 03 May 2016 16:10:59 GMT

We have a clustered environment that includes heavy forwarders, universal forwarders, and forwarders under Windows. The development team sometimes do performance tests and these generate a lot of data that we don't want to be indexed. We could add a new rule on the heavy forwarders to send to null queue all events during the tests , but can this be done at forwarder or universal forwarder level? Do you think that there is a better way to achieve this ?

Thank you

Re: What the best strategy to discard all temporary data while testing on some forwarders?

somesoni2 — Tue, 03 May 2016 16:52:09 GMT

Have a look at this Splunk documentation to know more about event routing and filter.
http://docs.splunk.com/Documentation/Splunk/6.4.0/Forwarding/Routeandfilterdatad#Filter_event_data_and_send_to_queues

The send to null queue can be done on universal forwarder if it's to be done without looking into individual events (purely based on index/source/sourcetype/host). If you need to look at the event data to filter, than you need to do routing/filtering in heavy forwarder/indexer

Re: What the best strategy to discard all temporary data while testing on some forwarders?

ddrillic — Tue, 03 May 2016 23:30:16 GMT

You can have the data indexed into specific indexes or add a specific field which indicates that this is a performance test data. Then it's easy to "simply" delete this type of data.

Re: What the best strategy to discard all temporary data while testing on some forwarders?

dsmc_adv — Wed, 04 May 2016 08:04:54 GMT

It looks like this only can be done at hf or indexer level as I suspected, but not in universal forwarder:

"Although similar to forwarder-based routing, queue routing can be performed by an indexer, as well as a heavy forwarder"