Splunk DB Connect ignoring configured intervals for database monitor

[johnebgood]

I have quite a few Oracle database inputs and I discovered they were running one query after another ignoring the interval. I initially had '60' for the interval and it was working initially as I watched the dbx.log like a hawk at first. I then swapped that out for '* * * * *' when I discovered the 60 stopped working and that worked on one indexer but on another I couldn't edit the properties via the UI and had to edit the inputs.conf to add the '* * * * *' cron syntax yet 2 of the 5 database monitor inputs continue to ignore the interval... ideas?

thanks,
John

[ziegfried]

There was a change how the interval value is interpreted in 1.0.6. Numbers are now treated as seconds (before they have been interpreted as milliseconds). See the release notes. You can also use relative time-expressions (eg. 1m for one minute).

The interval setting is not cached anywhere but is only read when the Java Bridge is restarted or reloaded. When you edit properties of a database input in the config file you typically have to restart Splunk. Database inputs are subject to Splunk's typical configuration layering. To view the actual settings in effect you can use btool:

$ splunk cmd btool inputs list dbmon --debug

Additionally you should be able to see all you configured database inputs and their schedule type when you take a look at the "Java Bridge Server Status" view in the Settings navigation menu in the app. Alternatively you can get the same status information in the console:

$ splunk cmd python $SPLUNK_HOME/etc/apps/dbx/bin/status.py

[gfuente]

Hello

I´m having exactly the same issue, did you find any way to solve it?

Regards

[johnebgood]

Setting that interval didn't work, it's still running some database monitors every 5 seconds.

In default/inputs.conf there is a script stanza:
[script]
interval = 60.0

I noticed that it was obeying the interval for the first monitor entry in dbx/local/inputs.conf the 2nd entry was a disabled entry and the rest were enabled but the interval wasn't being obeyed. I deleted the disabled entry and now different monitors are obeying the interval... going to troubleshoot some more.

[ziegfried]

You can add the interval to the [default] stanza in inputs.conf, this will affect scripted inputs as well, though.

[default]
interval = 60s

[johnebgood]

As a temporary fix is there a way I can change the default interval to 60 seconds?

[johnebgood]

btool:

search query = select * from table_name {{WHERE $rising_column$ > ?}}
dbx sourcetype = table_name
dbx table = table_name
dbx tail.rising.column = DATE
dbx [dbmon-tail://DATABASE/TABLE]
dbx disabled = 0
dbx host = db.domain.com
dbx index = index_name
dbx interval = * * * * *
dbx output.format = kv
dbx output.timestamp = 1
dbx output.timestamp.column = DATE

status.py:

[dbmon-tail://DATABASE/TABLE: valid=true disabled=false scheduleType=FIXED interval=5 running=false

[johnebgood]

I have tried cron format, 60s, etc and sporadically different input monitors don't obey the interval... other configured inputs with the same cron interval are currently working.

inputs.conf:

[dbmon-tail://DATABASE/TABLE]
host = db.domain.com
index = index_name
interval = * * * * *
output.format = kv
output.timestamp = 1
sourcetype = table_name
table = table_name
tail.rising.column = DATE
disabled = 0
output.timestamp.column = DATE

[johnebgood]

And one more discovery, the nextExecution for the inputs that are obeying the interval show the date and not a value:

Interval not working:
2013-01-24 20:27:58.911 main:DEBUG:Scheduler - Timer for input=[dbmon-tail://DATABASE/TABLE] nextExecution=5000 state=WAITING expired. Executing it now...

Interval working:
2013-01-24 20:28:00.000 main:DEBUG:Scheduler - Timer for input=[dbmon-tail://DATABASE2/TABLE] nextExecution=Thu Jan 24 20:28:00 CST 2013 state=WAITING expired. Executing it now...

[johnebgood]

Found a clue after turning on debugging, the nextExecution=5000, I'm tending to think this is 5 seconds. I initially configured the inputs for 5 seconds but later changed them to 60 seconds. Value being cached somewhere?

2013-01-24 20:24:27.722 main:DEBUG:Scheduler - Timer for input=[dbmon-tail://DATABASE/TABLE] nextExecution=5000 state=WAITING expired. Executing it now...

[johnebgood]

It's also sporadic, I updated to 1.0.7, restarted Splunk and now different databases are ignoring the interval.