Getting Data In

DB Connect tailing inputs fail after a Splunk restart

lukejadamec
Super Champion

Indexer – 2K8R2-64, Splunk 5.0.4, DB Connect 1.0.11, Latest JDK, ojdbc6,

Oracle 11 Server – 2K8R2-64

The problem is that after a Splunk restart (from manager, CLI, server power button) the tailing inputs fail.

This problem only occurs when there are two or more database connections either on the same Oracle server or one connection each on two different oracle servers.

There are two ways to recover from the problem:

1) Delete and recreate the inputs in DB Connect (this results in complete dump of the records causing duplicates)

2) (This is the weird one) Run a DB Connect query on one of the databases and kill Java in Task Manager, then re-run the query. All connections will start working until the next Splunk restart.

I’ve been trying to figure this out for a long time, and Splunk support has not yet found a solution.
It does not matter what order Java/Splunk are started, or how the server is restarted.

A DB Connect query must be running when Java is killed to restore the inputs.

There are no errors in any log at debug level. The logs indicate that everything is normal, but log sequences that would normally cycle to completion (scheduled tailing event for example) simply stop.

I have recreated this problem in two separate networks, but Splunk support has not been able to recreate it. Obviously it is me, what am I doing wrong?

Thanks

Tags (1)
0 Karma
1 Solution

ziegfried
Influencer

You should upgrade to the latest version of DB Connect. There is a specific fix that works around a JDBC deadlock issue, which might be what you're experiencing.

View solution in original post

ziegfried
Influencer

You should upgrade to the latest version of DB Connect. There is a specific fix that works around a JDBC deadlock issue, which might be what you're experiencing.

araitz
Splunk Employee
Splunk Employee

Can you open a support case so we can take a look at the diag?

0 Karma

lukejadamec
Super Champion

Running 1.0.11 and 5.0.5 now, and it still fails sometimes.

0 Karma

lukejadamec
Super Champion

Update, this deadlock fix does not work all the time.
I'm still running finding my Oracle inputs down after a reboot and reset with 1.0.11 and 5.0.4. Not always, but that makes it more of a pain because now it surprises me. When it was broken all the time it was not a surprise.

0 Karma

stefano_guidoba
Communicator

ehm we're running DB Connect on a RHEL x86_64 and the problem still shows with 1.0.11.
In fact we're using a single connection and 5 different inputs over 5 DB2 tables.
I've never contacted support because DB2 is in fact not supported, but Oracle instead is.

I guess the problem is related to the fact that when DBX JVM spawns, it tries to execute our 5 inputs simultaneously.

0 Karma

lukejadamec
Super Champion

I've found that the new version fixes the problem on 64bit, but not for 32bit OS//Splunk

0 Karma

lukejadamec
Super Champion

I tried the latest version, but it still is not working after a restart.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...