Getting Data In

Can shutting down an indexer break some forwarders; if so how to get them to resume.

RVDowning
Contributor

After upgrading from Splunk 5 to Splunk 6 some forwarders no longer forward data to the indexer. I haven't found a way to have them resume. I'm wondering if they were somehow "broken" by the shutdown process for the upgrade, namely left in some kind of inconsistent state.

Any idea how to find out which (of about 80) no longer forward and how to get them to resume?

Tags (2)
0 Karma

lukejadamec
Super Champion

This search will tell you the last time your forwarders were active - I run it for the past 7 days. Check the logs once you find the offline forwarders.

index=_internal source=*metrics.log group=tcpin_connections earliest=-7d@d 
| eval sourceHost=coalesce(hostname, sourceHost) 
| eval age = (now() - _time ) 
|stats first(age) as age,  first(_time) as LastTime by sourceHost 
| convert ctime(LastTime) as "Last Active On" 
| eval Status= case(age < XXX,"Running",age > XXX,"DOWN")

lukejadamec
Super Champion

I have seen cases where the easiest fix is to reinstall the forwarder, but of course it is always best to find the root cause when you can.

0 Karma

RVDowning
Contributor

Well, something new. The forwarder didn't work on 5.x so upgraded to 6.0 and still didn't work. Just went back to 5.x (also restarted the indexer) and now the data is being forwarded from that forwarder.

We still are not getting data from some other forwarders. Seems the trick it to what... Delete the 5.x forwarder and then reinstall it?

0 Karma

RVDowning
Contributor

Yes, the forwarders have been restarted. All forwards have the same outputs.conf. Am about to restore to 5.x.

0 Karma

lukejadamec
Super Champion

You have tried restarting those forwarders?
Are the outputs.conf the same for forwarders that are working/not working?

RVDowning
Contributor

Thanks for this; it is helpful. I see that the indexer server is refusing connections from a forwarder(s), but I can't seem to find any way to find out why.

0 Karma