Here's a quick rundown of the environment: Virtual Machines (linux splunk instances), No internet connection, air gapped environment that uses a unidirectional data diode. In this environment there is going to be very little data which is why there is just a single instance of Splunk (IDX, SH, and LM) and 1 universal forwarder. Oh, and for those of you reading along and new to splunk/networking and are asking "wtf is a data diode?" here is a short explanation "The concept of a data diode is simple: specifically designed hardware circuitry within which it is only possible for data to flow in one direction" In this case the data flows from the UF via UDP 514 to to side A interface of the diode with an example IP of (192.168.10.15). This interface is supposed to then push all of that forwarded data out of side B of the data diode which then pushes that data to the splunk server which is configured to listen on local input TCP 514 because I was told by the engineering team that's just how it was and didn't receive an explanation as to why one side was configured UDP and the other TCP.
The problem I have is ever since we added the diode aspect to the environment, Splunk no longer receives logs and I have no idea where to begin troubleshooting. The IP addresses in the UF and Splunk server have been corrected to reflect the change of location in the environment and rebinded the new IP address etc etc. Now, I don't know if this is because of a misconfiguration on my end of things or because the diode itself isn't properly setup yet. But from what I've explained am I understanding this is how the configuration is supposed to be in Splunk?
** Configure the universal forwarder to forward the syslog-ng data to the interface/IP of the data diode via UDP 514
** Then have the diode push that information outbound towards the splunk server
** Splunk is now listening on TCP 514 for the incoming syslog-ng data.
Side A of Diode (air gap) Side B of Diode
UF(x.x.10.25) -----> x.x.10.15 --> ||||| x.x.13.15 ----------> Splunk server(x.x.13.26)
Splunk should see this data as being sent from the data diode and not the universal forwarder correct? I would expect the logs to also include the IP addresses of both sides of the data diode as well as the IP of the UF..
Am I understanding this correctly? Or am I way off base?
... View more