After upgrading from Splunk 5 to Splunk 6 some forwarders no longer forward data to the indexer. I haven't found a way to have them resume. I'm wondering if they were somehow "broken" by the shutdown process for the upgrade, namely left in some kind of inconsistent state.
Any idea how to find out which (of about 80) no longer forward and how to get them to resume?
This search will tell you the last time your forwarders were active - I run it for the past 7 days. Check the logs once you find the offline forwarders.
index=_internal source=*metrics.log group=tcpin_connections earliest=-7d@d | eval sourceHost=coalesce(hostname, sourceHost) | eval age = (now() - _time ) |stats first(age) as age, first(_time) as LastTime by sourceHost | convert ctime(LastTime) as "Last Active On" | eval Status= case(age < XXX,"Running",age > XXX,"DOWN")
Well, something new. The forwarder didn't work on 5.x so upgraded to 6.0 and still didn't work. Just went back to 5.x (also restarted the indexer) and now the data is being forwarded from that forwarder.
We still are not getting data from some other forwarders. Seems the trick it to what... Delete the 5.x forwarder and then reinstall it?