All Apps and Add-ons

Splunk DB Connect: Inputs get disabled after server restarts. Is there a way to automatically re-enable the inputs or another solution?

johnpof
Path Finder

I work in a dynamic environment and often servers will get restarted for various reasons, updates for example..

Often I'll miss these events because I'm not on-call or nobody tells me. What happens is Splunk DB Connect disables all the inputs for that server and never tries to re-enable them.

So that data stops flowing into Splunk until someone manually re-enables the inputs for the servers in question.

My question is: Is there any way to have auto-retry logic? Or is there something I can use to monitor for inputs that get disabled? Right now I simply can't trust the data going into it because of this.

0 Karma
1 Solution

ppablo
Retired

Hi @johnpof

There's a previous Answers post similar to this topic with an answer that suggests disabling the auto_disable setting in inputs.conf. That might solve your issue:
https://answers.splunk.com/answers/438527/how-to-prevent-splunk-db-connect-2-from-disabling.html

View solution in original post

ppablo
Retired

Hi @johnpof

There's a previous Answers post similar to this topic with an answer that suggests disabling the auto_disable setting in inputs.conf. That might solve your issue:
https://answers.splunk.com/answers/438527/how-to-prevent-splunk-db-connect-2-from-disabling.html

hemendralodhi
Contributor

We have SHC environment and even after setting auto_disable = false for input it is getting disabled whenever SHC gets restarted? How to prevent it from disabling then? Here is my configuration:

[mi_input://Fusion_DG_Lag]
connection = Fusion_Common_DB_PRD
description = Lag between Primary and Standby DB
enable_query_wrapping = 1
index = index
input_timestamp_column_fullname = (003) NULL.TIME_COMPUTED.VARCHAR2
input_timestamp_column_name =
interval = 120
max_rows = 10000
mode = batch
output_timestamp_format = dd-MM-yyyy HH:mm:ss
query = SELECT name,\
((SUBSTR(value,2,2)*86400)+(SUBSTR(value,5,2)*3600)+(SUBSTR(value,8,2)*60)+(SUBSTR(value,11,2))) lag_time,\
time_computed\
FROM V$DATAGUARD_STATS\
WHERE name IN ('apply lag','transport lag')
source = fusion_common_db_prd
sourcetype = fusion_db
tail_rising_column_fullname = (003) NULL.TIME_COMPUTED.VARCHAR2
tail_rising_column_name = TIME_COMPUTED
ui_query_catalog = NULL
ui_query_mode = advanced
auto_disable = false

0 Karma

johnpof
Path Finder

I'm pretty sure DBX 2.0 is not supported in an SHC environment, I had all sorts of problems with it and eventually moved it to a standalone search head by itself outside of my SHC.

0 Karma

hemendralodhi
Contributor

I am running V 2.3.1 which support SHC but it is in documentation that running input/output is not recommended in SHC. I overlooked this and probably need to run it in standalone instance.

0 Karma

johnpof
Path Finder

So I came in this morning after a major maintenance, every single input was disabled.

so unfortunately this isn't working 😕 does it matter where you put the auto_disable = False?

0 Karma

ppablo
Retired

Hi @johnpof

Sorry I just saw your follow up comment. Did you happen to figure it out? I noticed the value you used for auto_disable was "False", not "false" as shown in the documentation. I'm not sure if the value is case sensitive.

0 Karma

johnpof
Path Finder

I noticed that too actually just yesterday and replaced everything 😛

I bet it'll be fine now, thanks for follow up.

0 Karma

johnpof
Path Finder

For future reference, I used this handy line to add auto_disable = False right after all occurrences of disabled = 0

sed -i $'s/disabled = 0/disabled = 0\\\nauto_disable = False/g' inputs.conf

ppablo
Retired

Thanks for sharing that @johnpof 🙂 Cheers!

0 Karma

johnpof
Path Finder

Nice! that's perfect. Can I update inputs.conf live and will the changes immediately be reflected?

0 Karma

koshyk
Super Champion

can u try updating the inputs.conf using REST API. if you can find it, then it doesn't require restart hopefully

0 Karma

ppablo
Retired

Hm, I'm not positive about that actually. I just remembered coming by that previous question before and it came to mind when I saw yours.

From looking at the inputs.conf spec for Splunk Enterprise docs, it says Splunk has to be restarted for config changes to take effect:
http://docs.splunk.com/Documentation/Splunk/6.5.0/admin/Inputsconf

I don't see that specifically in the DB Connect inputs.conf docs, but I would assume that a restart is still required after any changes:
http://docs.splunk.com/Documentation/DBX/2.3.0/DeployDBX/inputsspec

Sorry I don't have a definitive answer, but hopefully someone who does have experience with this can confirm.

johnpof
Path Finder

I'll probably restart it and see what happens

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...