I am looking to set up a monitoring tool (HP's Sitescope) to "watch" our forwarders to ensure they are up. I am not sure specifically what I should watch in the forwarder (port, process, etc). It seems the outgoing port changes after each restart, and I'm not sure that monitoring the Management Host Port (8089) would provide an accurate indication if the forwarder stopped working.
Running Splunk 4.1.2.. I know 4.2 has forwarder monitoring built in, but I'm looking for an interim solution.