We have a database data input which has stopped pulling data from an oracle database over the last 7-8 hours. I have tried disabling and re enabling the data inputs to no effect. i have also gone to manage apps> splunk DB connect>view objects and disabled and re enabled the object related to that database but no luck there either.
Then i tried to restart the splunk services on the indexers, one of them started fine but the other one keeps going into upgrade setup every time we try to start or stop it (it mistakes the start/stop for upgrade request, I presume).
Has anyone faced this forced upgrade issue before? How to troubleshoot it?
Do you think the two issues are related? Will restarting services on the indexer solve the first issue?
Also we tried to create a diagnosis file on the indexer server, for adding to a splunk.com case I have already raised, but even the diag command is making the upgrade setup open up and we are unable to capture the diag file.