Can shutting down an indexer break some forwarders; if so how to get them to resume.

[RVDowning]

After upgrading from Splunk 5 to Splunk 6 some forwarders no longer forward data to the indexer. I haven't found a way to have them resume. I'm wondering if they were somehow "broken" by the shutdown process for the upgrade, namely left in some kind of inconsistent state.

Any idea how to find out which (of about 80) no longer forward and how to get them to resume?

[lukejadamec]

This search will tell you the last time your forwarders were active - I run it for the past 7 days. Check the logs once you find the offline forwarders.

index=_internal source=*metrics.log group=tcpin_connections earliest=-7d@d 
| eval sourceHost=coalesce(hostname, sourceHost) 
| eval age = (now() - _time ) 
|stats first(age) as age,  first(_time) as LastTime by sourceHost 
| convert ctime(LastTime) as "Last Active On" 
| eval Status= case(age < XXX,"Running",age > XXX,"DOWN")

[lukejadamec]

I have seen cases where the easiest fix is to reinstall the forwarder, but of course it is always best to find the root cause when you can.

[RVDowning]

Well, something new. The forwarder didn't work on 5.x so upgraded to 6.0 and still didn't work. Just went back to 5.x (also restarted the indexer) and now the data is being forwarded from that forwarder.

We still are not getting data from some other forwarders. Seems the trick it to what... Delete the 5.x forwarder and then reinstall it?

[RVDowning]

Yes, the forwarders have been restarted. All forwards have the same outputs.conf. Am about to restore to 5.x.

[lukejadamec]

You have tried restarting those forwarders?
Are the outputs.conf the same for forwarders that are working/not working?

[RVDowning]

Thanks for this; it is helpful. I see that the indexer server is refusing connections from a forwarder(s), but I can't seem to find any way to find out why.