How to drop _internal logs received from universal forwarders on a heavy forwarder?

[thezero]

Hi Team,

We need to drop _internal logs forwarded by universal forwarders as _internal logs are consuming most of the disk space. As the number of universal forwarders is high, it's not possible to change configs on the universal forwarder. Could you please advise on how can we stop indexing _internal received from universal forwarders? How can we drop them on heavy weight forwarder? We just want to enable _internal logs indexing for the heavy weight forwarder but not for Universal forwarders. Please advise.

Our Log flow:

Universal forwarder ---> Heavy weight forwarder --->Indexer

[jeffland · Accepted solution]

As stephanefotso noted, you should change your forwarders to stop sending those instead of dropping them on your heavy forwarders, it saves processing and bandwidth. You should use a deployment server and forwarder management if your environment is at a size where you don't want to edit .conf files on your forwarders manually. I'd generally recommend that anyway to make sure your settings are consistent and managed centrally.

If you're worried about the size of your _internal index, you could also consider changing the retention settings for those. Having them for the last few days is probably useful, but you may not need to keep them for 30 days (which is the standard).

[thezero]

Hi sephen/Jeff,

Thanks for your advice.Actually we have few thousands universal forwarders deployed and configuring all of them - to not to send data is not practical (considering their numbers).Also deployment server is configured to perform deployments for heavy weight forwarders only.So trying to figure out the way to do it on heavy weight forwarder only.Let me know if it is possible to drop _internal logs received from universal forwarders on HWF fusing filtering or any other solution?

[jeffland]

You have a few thousand forwarders configured WITHOUT deployment server? Wow.
By default, your universal forwarders do not forward their _internal index data; only heavy forwarders do. They forward their _audit index however. Someone must have enabled this on all your UFs. I'd say the easiest thing to do is enable deployment for all your forwarders and change it that way.
You could disable sending _internal from your heavy forwarders altogether, but that includes your heavy forwarders _internal index as well. You can also try and see if you can get a regex working to filter the data, but that really just creates unnecessary load on your heavy forwarders.

[stephanefotso]

Hi. You must edit outputs.conf configuration file on your forwarders. Read "Filter data by target index" here http://docs.splunk.com/Documentation/Splunk/6.3.3/Forwarding/Routeandfilterdatad

If any question, let know

Thanks