We have a database data input which has stopped pulling data from an oracle database over the last 7-8 hours. I have tried disabling and re enabling the data inputs to no effect. i have also gone to manage apps> splunk DB connect>view objects and disabled and re enabled the object related to that database but no luck there either.
Then i tried to restart the splunk services on the indexers, one of them started fine but the other one keeps going into upgrade setup every time we try to start or stop it (it mistakes the start/stop for upgrade request, I presume).
For this sort of urgent issue, it's probably better to call support... I'm not seeing which version of DB Connect or Splunk you're using, which would definitely affect the potential answer.
Also we tried to create a diagnosis file on the indexer server, for adding to a splunk.com case I have already raised, but even the diag command is making the upgrade setup open up and we are unable to capture the diag file.