I work in a dynamic environment and often servers will get restarted for various reasons, updates for example..
Often I'll miss these events because I'm not on-call or nobody tells me. What happens is Splunk DB Connect disables all the inputs for that server and never tries to re-enable them.
So that data stops flowing into Splunk until someone manually re-enables the inputs for the servers in question.
My question is: Is there any way to have auto-retry logic? Or is there something I can use to monitor for inputs that get disabled? Right now I simply can't trust the data going into it because of this.
There's a previous Answers post similar to this topic with an answer that suggests disabling the auto_disable setting in inputs.conf. That might solve your issue:
Nice! that's perfect. Can I update inputs.conf live and will the changes immediately be reflected?
Hm, I'm not positive about that actually. I just remembered coming by that previous question before and it came to mind when I saw yours.
From looking at the inputs.conf spec for Splunk Enterprise docs, it says Splunk has to be restarted for config changes to take effect:
I don't see that specifically in the DB Connect inputs.conf docs, but I would assume that a restart is still required after any changes:
Sorry I don't have a definitive answer, but hopefully someone who does have experience with this can confirm.
I'll probably restart it and see what happens
can u try updating the inputs.conf using REST API. if you can find it, then it doesn't require restart hopefully
For future reference, I used this handy line to add auto_disable = False right after all occurrences of disabled = 0
sed -i $'s/disabled = 0/disabled = 0\\\nauto_disable = False/g' inputs.conf
Thanks for sharing that @johnpof 🙂 Cheers!
So I came in this morning after a major maintenance, every single input was disabled.
so unfortunately this isn't working 😕 does it matter where you put the auto_disable = False?
Sorry I just saw your follow up comment. Did you happen to figure it out? I noticed the value you used for auto_disable was "False", not "false" as shown in the documentation. I'm not sure if the value is case sensitive.