Say for example i have a environment that is being split into two different networks (routed through different firewalls) I have my "main" splunk server on one of the network. Is is possible to use one or more splunk instance as "collectors" on the other net and send it to the main splunk server. I rather don't want to set up routing/firewalls opening more then absolutely necessary.
You can absolutely set up a firewall between separate subnets, but as with anything IP based, you're going to need firewall rules to allow it. If you set up a forwarder in subnet A, it's going to forward to whatever port you define in your inputs.conf on the indexer (and outputs.conf on the forwarder). You can set whatever port you want, though, so it will you don't need to allow TCP Any, or port 80/443, or etc. You could, say, use port 12345, which is nice and distinct for firewall admins.
Ah ha, gotcha. Unless you're sending obscenely large amounts of data, or have an obscenely underpowered box in Net A, that should work fine. Forwarding is very resource light, so you can have a local forwarder send everything to the main box. Just note that you have to have the box in Net A be a forwarder, rather than an indexer. You can't double-index the data, unless you have the licensing to support it.
Yeah well im also doing a big part of the firewall stuff myself so i know about that stuff. What i meant was if its working good to have some servers on Net A send their stuff to a smaller splunk server on net A, that server then sends it stuff to the "real" splunk server on net B. With that i only have the static route between the two splunk servers configured, insted of the hole net A- Net B.