Splunk DB Connect: Inputs get disabled after server restarts. Is there a way to automatically re-enable the inputs or another solution?

[johnpof]

I work in a dynamic environment and often servers will get restarted for various reasons, updates for example..

Often I'll miss these events because I'm not on-call or nobody tells me. What happens is Splunk DB Connect disables all the inputs for that server and never tries to re-enable them.

So that data stops flowing into Splunk until someone manually re-enables the inputs for the servers in question.

My question is: Is there any way to have auto-retry logic? Or is there something I can use to monitor for inputs that get disabled? Right now I simply can't trust the data going into it because of this.

[ppablo · Accepted solution]

Hi @johnpof

There's a previous Answers post similar to this topic with an answer that suggests disabling the auto_disable setting in inputs.conf. That might solve your issue:
https://answers.splunk.com/answers/438527/how-to-prevent-splunk-db-connect-2-from-disabling.html

[hemendralodhi]

We have SHC environment and even after setting auto_disable = false for input it is getting disabled whenever SHC gets restarted? How to prevent it from disabling then? Here is my configuration:

[mi_input://Fusion_DG_Lag]
connection = Fusion_Common_DB_PRD
description = Lag between Primary and Standby DB
enable_query_wrapping = 1
index = index
input_timestamp_column_fullname = (003) NULL.TIME_COMPUTED.VARCHAR2
input_timestamp_column_name =
interval = 120
max_rows = 10000
mode = batch
output_timestamp_format = dd-MM-yyyy HH:mm:ss
query = SELECT name,\
((SUBSTR(value,2,2)*86400)+(SUBSTR(value,5,2)*3600)+(SUBSTR(value,8,2)*60)+(SUBSTR(value,11,2))) lag_time,\
time_computed\
FROM V$DATAGUARD_STATS\
WHERE name IN ('apply lag','transport lag')
source = fusion_common_db_prd
sourcetype = fusion_db
tail_rising_column_fullname = (003) NULL.TIME_COMPUTED.VARCHAR2
tail_rising_column_name = TIME_COMPUTED
ui_query_catalog = NULL
ui_query_mode = advanced
auto_disable = false

[johnpof]

I'm pretty sure DBX 2.0 is not supported in an SHC environment, I had all sorts of problems with it and eventually moved it to a standalone search head by itself outside of my SHC.

[hemendralodhi]

I am running V 2.3.1 which support SHC but it is in documentation that running input/output is not recommended in SHC. I overlooked this and probably need to run it in standalone instance.

[johnpof]

So I came in this morning after a major maintenance, every single input was disabled.

so unfortunately this isn't working 😕 does it matter where you put the auto_disable = False?

[ppablo]

Hi @johnpof

Sorry I just saw your follow up comment. Did you happen to figure it out? I noticed the value you used for auto_disable was "False", not "false" as shown in the documentation. I'm not sure if the value is case sensitive.

[johnpof]

I noticed that too actually just yesterday and replaced everything 😛

I bet it'll be fine now, thanks for follow up.

[johnpof]

For future reference, I used this handy line to add auto_disable = False right after all occurrences of disabled = 0

sed -i $'s/disabled = 0/disabled = 0\\\nauto_disable = False/g' inputs.conf

[ppablo]

Thanks for sharing that @johnpof 🙂 Cheers!

[johnpof]

Nice! that's perfect. Can I update inputs.conf live and will the changes immediately be reflected?

[koshyk]

can u try updating the inputs.conf using REST API. if you can find it, then it doesn't require restart hopefully

[ppablo]

Hm, I'm not positive about that actually. I just remembered coming by that previous question before and it came to mind when I saw yours.

From looking at the inputs.conf spec for Splunk Enterprise docs, it says Splunk has to be restarted for config changes to take effect:
http://docs.splunk.com/Documentation/Splunk/6.5.0/admin/Inputsconf

I don't see that specifically in the DB Connect inputs.conf docs, but I would assume that a restart is still required after any changes:
http://docs.splunk.com/Documentation/DBX/2.3.0/DeployDBX/inputsspec

Sorry I don't have a definitive answer, but hopefully someone who does have experience with this can confirm.

[johnpof]

I'll probably restart it and see what happens