I am looking to set up a monitoring tool (HP's Sitescope) to "watch" our forwarders to ensure they are up. I am not sure specifically what I should watch in the forwarder (port, process, etc). It seems the outgoing port changes after each restart, and I'm not sure that monitoring the Management Host Port (8089) would provide an accurate indication if the forwarder stopped working.
Running Splunk 4.1.2.. I know 4.2 has forwarder monitoring built in, but I'm looking for an interim solution.
Thank you,
DevonK
i know you're thinking of using an external solution, but you could also set up an alert using this search:
| metadata type=hosts | eval age = now() - lastTime | search age > 86400 | sort age d | convert ctime(lastTime) | fields age,host,lastTime
to show you forwarders that haven't been heard from in the last 24 hours.