I work in a dynamic environment and often servers will get restarted for various reasons, updates for example..
Often I'll miss these events because I'm not on-call or nobody tells me. What happens is Splunk DB Connect disables all the inputs for that server and never tries to re-enable them.
So that data stops flowing into Splunk until someone manually re-enables the inputs for the servers in question.
My question is: Is there any way to have auto-retry logic? Or is there something I can use to monitor for inputs that get disabled? Right now I simply can't trust the data going into it because of this.
Hi @johnpof
There's a previous Answers post similar to this topic with an answer that suggests disabling the auto_disable setting in inputs.conf. That might solve your issue:
https://answers.splunk.com/answers/438527/how-to-prevent-splunk-db-connect-2-from-disabling.html
Hi @johnpof
There's a previous Answers post similar to this topic with an answer that suggests disabling the auto_disable setting in inputs.conf. That might solve your issue:
https://answers.splunk.com/answers/438527/how-to-prevent-splunk-db-connect-2-from-disabling.html
We have SHC environment and even after setting auto_disable = false for input it is getting disabled whenever SHC gets restarted? How to prevent it from disabling then? Here is my configuration:
[mi_input://Fusion_DG_Lag]
connection = Fusion_Common_DB_PRD
description = Lag between Primary and Standby DB
enable_query_wrapping = 1
index = index
input_timestamp_column_fullname = (003) NULL.TIME_COMPUTED.VARCHAR2
input_timestamp_column_name =
interval = 120
max_rows = 10000
mode = batch
output_timestamp_format = dd-MM-yyyy HH:mm:ss
query = SELECT name,\
((SUBSTR(value,2,2)*86400)+(SUBSTR(value,5,2)*3600)+(SUBSTR(value,8,2)*60)+(SUBSTR(value,11,2))) lag_time,\
time_computed\
FROM V$DATAGUARD_STATS\
WHERE name IN ('apply lag','transport lag')
source = fusion_common_db_prd
sourcetype = fusion_db
tail_rising_column_fullname = (003) NULL.TIME_COMPUTED.VARCHAR2
tail_rising_column_name = TIME_COMPUTED
ui_query_catalog = NULL
ui_query_mode = advanced
auto_disable = false
I'm pretty sure DBX 2.0 is not supported in an SHC environment, I had all sorts of problems with it and eventually moved it to a standalone search head by itself outside of my SHC.
I am running V 2.3.1 which support SHC but it is in documentation that running input/output is not recommended in SHC. I overlooked this and probably need to run it in standalone instance.
So I came in this morning after a major maintenance, every single input was disabled.
so unfortunately this isn't working 😕 does it matter where you put the auto_disable = False?
Hi @johnpof
Sorry I just saw your follow up comment. Did you happen to figure it out? I noticed the value you used for auto_disable was "False", not "false" as shown in the documentation. I'm not sure if the value is case sensitive.
I noticed that too actually just yesterday and replaced everything 😛
I bet it'll be fine now, thanks for follow up.
For future reference, I used this handy line to add auto_disable = False right after all occurrences of disabled = 0
sed -i $'s/disabled = 0/disabled = 0\\\nauto_disable = False/g' inputs.conf
Thanks for sharing that @johnpof 🙂 Cheers!
Nice! that's perfect. Can I update inputs.conf live and will the changes immediately be reflected?
can u try updating the inputs.conf using REST API. if you can find it, then it doesn't require restart hopefully
Hm, I'm not positive about that actually. I just remembered coming by that previous question before and it came to mind when I saw yours.
From looking at the inputs.conf spec for Splunk Enterprise docs, it says Splunk has to be restarted for config changes to take effect:
http://docs.splunk.com/Documentation/Splunk/6.5.0/admin/Inputsconf
I don't see that specifically in the DB Connect inputs.conf docs, but I would assume that a restart is still required after any changes:
http://docs.splunk.com/Documentation/DBX/2.3.0/DeployDBX/inputsspec
Sorry I don't have a definitive answer, but hopefully someone who does have experience with this can confirm.
I'll probably restart it and see what happens