How to forward to an intermediate forwarder and then to an indexer, while also getting deployment apps via the intermediate forwarder from a deployment server.

[wrangler2x]

We have a number of systems inside of a special, firewalled-off network we call FacNet, which is a subnet inside of a larger class B network. The firewall does not allow connections to be established from systems on FacNet to anything outside of that network, with the sole exception of a DMZ-based Linux SMTP proxy. A recent audit recommended these systems forward their logs to our campus Splunk instance (Indexer). The folks in control of the firewall say they cannot permit connections to the Splunk Indexer or Deployment Server.

So I am thinking that the systems inside of FacNet could be simply configured to send logs to a DMZ-based heavy forwarder and in every other way be configured as any other of our forwarders. And it seems easy enough to configure the DMZ-based heavy forwarder to send logs to the indexer. But what I don't know how to configure is how the DMZ-based heavy forwarder is to be configured on the input side, and how to tell it to forward the logs it receives to the indexer.

As described above, this is problem number one, and I'm looking for someone who knows how to do this to give me some help or pointers to documentation that explains it.

Problem number two is how the forwarders inside of FacNet can get their deployment apps from the Deployment Server. Assuming this is even possible, how do you configure the DMZ-based heavy forwarder to act as a proxy between the Deployment server outside of FacNet and the forwarders inside of it? Has anyone done this?

[hagjos43]

We've done something similar in our environment with the exception of deployment server. I suppose you could set up a second deployment server within your semi-air-gapped network then you'd only have to manage configurations from that point. I'll have to think on that one for a bit.

As far as forwarding logs from your heavy forward to a specified indexer that's easy and you can do it all in the GUI!
Our environment looks like this:
Heavy Forwarder > Indexer1 (indexer one is configured to replicate to our other indexers that are part of an index cluster)

To configure this in the GUI simply follow these steps:
1. Log onto the heavy forwarder web console
2. Settings > Forwarding and Receiving > Configure Forwarding > Select All data from all apps (not sure if this matters > click "new" > enter IP:PORT

That's it. Obviously make sure your indexer is configured to receive data on whatever port you specify, and the firewall folks have allowed data out of the DMZ over said port and between IP addresses.

Hope that's what you were looking for and hope that helps!

[wrangler2x]

When you do that (use the GUI to configure forwarding) I assume it sets the [license] stanza in server.conf to be active_group = Forwarder, correct? But also I'll need to use SSL on the DMZ forwarder and send to port 9998 on the indexer (because that is how the indexer is configured to receive), so I'll have to configure that in outputs.conf on the DMZ forwarder...

I'm supposing that I can use the inputs.conf settings I use on the indexer on the DMZ forwarder. That sound right?

As to the idea of having a second deployment server running on the DMZ forwarder, I'd thought about that and that is doable, except that I'm being given temporary admin access to the DMZ box (it is not mine) and would have to request that every time I needed to make a change to the deployment apps. Not the end of the world, but it sure would be nice to have one copy of all the deployment apps in one place, on a box I have admin access to (which I do on my existing deployment server). So that is why I am interested in seeing if there is a way to proxy it via the DMZ forwarder.