our network count ~9000 Servers. Most of them running in the separate network IP segments. I would like to kindly ask You about log forwarding from that machines. Between indexer and some servers we have to build several hops (forwarders). How to build it properly ?
Please take a look on that example:
Linux Server GROUP LAN1 (splunk forwarder which one ?) ----- > Splunk Forwarder (which one ?) LAN1 ----> Splunk Indexer
Linux Server GROUP LAN4 ----> splunk forwarder LAN3 ---....---> Splunk Forwarder LAN1 --- > Splunk Indexer ?
If I good understand Splunk architecture, on the each machine I have to install Splunk Universal Forwards(or lightforwarder ?) to transfer logs from the local running applications. Each Universal Forwarder installed on the app servers will push the logs to the
heave forwarder which will be connected with the next hop (also Heave Forwarder or in the final step with the indexer). Is this the proper solution ?
lightforwarder ---> heave forwarder ---> heavy forwarder ----> Indexer ?
What about loadbalancing, we need it. If we would like to push logs from ~500 heavy loaded systems, we need minimum two machines I suspect. Is it possible to loadbalance such a traffic ?
Thanks in advance for any hints.
With kind regards
I do not know if I need this 😉 this was only question if its necessary.
We will transfer logs from different machines, devices and I thought that it would be better to build Heavy Forwarders (with a lot of options to filter logs) between.
What You prupose ?
My advice is to keep things simple. Universal forwards are designed to communicate perfectly fine with the indexers. So, if you don't have at the moment any special parsing needs I would go with the simplest approach.
At the moment we have no any special parsing requirements but if its possible we would like to parse logs on our hops/forwarders. Which forwarder will be the best one for it ?
Our target is to keep application server configuration as simple as possible so we will forward everything to the first forwarder, where we would like to setup filtering for an each incoming log stream. In that case Heavy Forwarder make sense, isnt it ?
Look, you can filter at the heavy forwarder level or at the indexer level. Normally, we start with the simplest architecture and then amplify it based on the needs, but obviously, you can start with more evolved architecture. When we do have heavy forwarders, it's normally in order to alleviate pressure from the indexers, so if you know about a high velocity stream of data that needs to be parsed, then the heavy forwarders layer(s) might make sense.
This would be an extremely complex environment, and very difficult to maintain as an administrator.
The only reason I've ever seen this done, is not because of VLAN's or segments, but data centers and cost. Basically, the data center charged the company I was working with per segment it opened, and per host, due to NAT'ing all traffic at the edge. opening a hole in the firewall for the ~1200 machines behind it was cost prohibitive.
I would agree with everything below, in that simple is better. you have 9000+ servers. I can say that I have administered 8300+ servers, and used a single heavy forwarder in each data center (in that case there were 5 datacenters) to do any excess pre-index parsing necessary. Mind you that includes running 5000+ nodes of hadoop parsing through them.
making this too complicated will hurt your efforts in the future, depending on the competency, and number of your staff that is familiar with Splunk. if it's just you... this would be more pain than it's worth.
for your load balancing efforts, simply add
forceTimeBasedLoadBalancing = true to your outputs.conf on any forwarder, and then
autoLB = true. This should take care of equally load balancing your data across your indexers.
look them up in the docs for outputs.conf
Your forwarding architecture, while complex and difficult to troubleshoot, is technically workable. Sometimes, you have to do what you have to do in a highly segmented network with overly(?) restrictive firewall policies.
Unless you have advanced routing/filtering requirements, you don't need a heavy forwarder anywhere; universal forwarders can do it, and do it more efficiently.
You need to take care not to introduce any bottleneck that will affect your event distribution across the indexer tier. The last forwarding tier needs to have at least 2x forwarders (or ingestionPipelines) than you have indexers to ensure indexers are served concurrently and as evenly as possible.
Your first intermediary tier needs to be properly sized as well to handle the 9000 endpoints (2 TCP connections each, one heartbeat, one data). Make sure you properly set your ulimits such that your intermediary forwarders can handle the large connection counts.
If your intermediary forwarder servers are equipped with enough cores, you can safely increase parallelIngestionPipelines to have each forwarder process multiple inbound/outbound connections in parallel. See the documentation for details. If you have eight-core servers, you can safely configure four parallel pipelines.
Also, ensure you have sufficient network bandwidth on ingress and egress to handle the data volume going through your intermediary forwarding tier. And don't introduce single points of failure into your intermediary forwarding architecture, i.e. have at least 2 servers at each tier.
Hope that helps.