We have requirement to add a Heavy Forwarder tier between Universal Forwarder and Indexers.
Is there a recommended port for communication between UF -> HF?
I know that port 9997 can be used for communication between HF -> IDX.
I am aware that all the above ports are configurable and just wanted to know if there are any recommendations and best practices while setting intermediate forwarding.
haven't seen any best practices around it as well but i think its very safe to keep it at default ports on HF for
have seen it like that in many environments.
best practice is to avoid using an intermediate forwarder between forwarders and indexers
hope it helps
My requirement is to filter events at the heavy forwarder, before I send them to the Indexer.
So can both communication happen on 9997?
UF -> HF - 9997
HF -> IDX - 9997
Also, if I don't use a heavy forwarder and do the filtering at Indexers, will it consume any license?
Given all the details you provided (about filtering), I would encourage a slightly different approach, but first, to answer your question.
There is no recommendation around ports. You are welcome to use the default ones. If you prefer security through obscurity, you can use alternate ones. All ports are treated equally so there's not difference there (assuming Splunk is not running as root, you'll be limited to a port above 1024 due to OS restrictions on Unix).
BUT, don't worry about that because the impact of adding a HF in your topology is devastating compared to letting the indexers do the filtering directly. To elaborate:
index=_internal source=*license_usage* type=Usage.
That was a lot of stream-of-brain pre-coffee so shout if any of it is unclear.
Thanks Slosh for the detailed response! Just one last question.
If I am doing a lot of filtering at Indexer level will it impact the Indexer performance (increased CPU or indexing lag)?
Yes. Even a pebble dropping in the air makes a slight impact the wind around it. (how's that for zen!)
Unfortunately, I can't think of a confident means to measure the impact but I would gamble that unless we're talking about regex on every single event and regex that's really poorly written then you may not even be able to notice the difference.
The worst that happens is you make a business case for more indexers, which means you're searching will perform that much better. So yea, there's a cost but there's also benefits. You'll have to decide which approach produces the best net positive for you when everything is taken into account. Fair?