DB Connect tailing inputs fail after a Splunk restart

[lukejadamec]

Indexer – 2K8R2-64, Splunk 5.0.4, DB Connect 1.0.11, Latest JDK, ojdbc6,

Oracle 11 Server – 2K8R2-64

The problem is that after a Splunk restart (from manager, CLI, server power button) the tailing inputs fail.

This problem only occurs when there are two or more database connections either on the same Oracle server or one connection each on two different oracle servers.

There are two ways to recover from the problem:

1) Delete and recreate the inputs in DB Connect (this results in complete dump of the records causing duplicates)

2) (This is the weird one) Run a DB Connect query on one of the databases and kill Java in Task Manager, then re-run the query. All connections will start working until the next Splunk restart.

I’ve been trying to figure this out for a long time, and Splunk support has not yet found a solution.
It does not matter what order Java/Splunk are started, or how the server is restarted.

A DB Connect query must be running when Java is killed to restore the inputs.

There are no errors in any log at debug level. The logs indicate that everything is normal, but log sequences that would normally cycle to completion (scheduled tailing event for example) simply stop.

I have recreated this problem in two separate networks, but Splunk support has not been able to recreate it. Obviously it is me, what am I doing wrong?

Thanks

[ziegfried · Accepted solution]

You should upgrade to the latest version of DB Connect. There is a specific fix that works around a JDBC deadlock issue, which might be what you're experiencing.

[araitz]

Can you open a support case so we can take a look at the diag?

[lukejadamec]

Running 1.0.11 and 5.0.5 now, and it still fails sometimes.

[lukejadamec]

Update, this deadlock fix does not work all the time.
I'm still running finding my Oracle inputs down after a reboot and reset with 1.0.11 and 5.0.4. Not always, but that makes it more of a pain because now it surprises me. When it was broken all the time it was not a surprise.

[stefano_guidoba]

ehm we're running DB Connect on a RHEL x86_64 and the problem still shows with 1.0.11.
In fact we're using a single connection and 5 different inputs over 5 DB2 tables.
I've never contacted support because DB2 is in fact not supported, but Oracle instead is.

I guess the problem is related to the fact that when DBX JVM spawns, it tries to execute our 5 inputs simultaneously.

[lukejadamec]

I've found that the new version fixes the problem on 64bit, but not for 32bit OS//Splunk

[lukejadamec]

I tried the latest version, but it still is not working after a restart.