Heartbeat for forwarder

[sdevadas]

Hi,

I am trying to setup a heartbeat to know if our Splunk forwarders are working fine.
From this forum, I found I can easily query for forwarders which have been quiet while eliminating very old (say retired) forwarders.
e.g., query for forwarders which have been quiet for 1 hour upto a max of 1 day:
| metadata type=hosts | eval age = now() - lastTime | search (age > 3600 AND age < 86400) | sort age d | convert ctime(lastTime) | fields age, host, lastTime

I would like to now setup a heartbeat event from the forwarder. Is this possible using Splunk forwarder configuration?
I saw the sendCookedData and heartbeatFrequency settings for outputs.conf of a forwarder.
e.g.:
sendCookedData = true
heartbeatFrequency = 30

Haven't tested this out yet but, will these settings actually send an event every 30 seconds to the indexer, so that the search given before will omit such forwarders even if they are not emitting any other events?
If this works, are there any other side effects of these settings, which I should be aware of?

If this does not work and the heartbeat does not generate any events, is there any way I can generate heartbeat events from within the Splunk forwarder?

Thanks

[sdevadas]

gkanapathy, Thanks again. Here's the search I ended up alerting on:

| set diff [search earliest="-30m" index=_internal host="MYINDEXERS*" source="/opt/splunk/var/log/splunk/metrics.log" | dedup hostname | table hostname] [search earliest="-10m" index=_internal host="MYINDEXERS*" source="/opt/splunk/var/log/splunk/metrics.log" | dedup hostname | table hostname]

[rmorlen]

I tried this but it didn't work for me. (Running Splunk 4.x). Substituted MYINDEXERS with our indexer and use "*metrics*" instead of what you have. Still didn't work.

[gkanapathy]

The Splunk indexers already report on every forwarder connection every 30 seconds in the internal metrics log.

if you really want, the forwarders themselves also write into their own log internal metrics logs every 30 seconds at least. These, however, are by default not forwarded as of most recent versions of Splunk.

[sdevadas]

Thanks, this is helpful. I had turned off the indexer logs indexing in all envs, will to turn it on and test it out in QA environment.