How to track successful inbound connections to a universal forwarder?

[djconroy]

I have a group of Universal forwarders deployed in our DMZ to relay logs from UF's in the field to our indexing cluster inside our network.

The splunkd.log from the DMZ servers shows errors on those inbound connections when they occur, but it does not appear to log successful inbound connections, which would be helpful in performance tracking and iplocation of clients.

It does seem to capture successful outbound connections to the indexers, so I would have to imaging this information is available.

Is there a logging level switch somewhere that would trap that information in the splunkd.log in the DMZ?

[djconroy · Accepted solution]

So I figured this one out on my own... it appears you can glean that information from the metrics logs in /var/log/splunk. For whatever reason, those logs, although configured to be monitored in /etc/system/defaults/inputs.conf in the Universal Forwarders, were not being forwarded or indexed at the cluster level. I added the folder specifically to the local/inputs.conf, and added a dedicated index and input/output for those logs from the DMZ for easy searching of inbound statistics from our internet-based forwarders.

[djconroy]

I later learned that you can allow the default inputs.conf to pick them up those metrics if you add a the following to the tcpout stanza in outputs.conf:

[tcpout]
forwardedindex.filter.disable = true

This prevents you from having to index the metrics from the forwarders themselves against your license.