All Apps and Add-ons

How to install and configure Splunk DB Connect 2.0.5 in a Splunk 6.3.0 environment with indexer clustering, but no search head clustering?

napomokoetle
Communicator

Hi folks,

I have installed Splunk 6.3 on Centos Linux 6.5

The installation consists of...
- a dedicated Search Head
- a master node that also serves as a Search Head
- two clustered indexers
- two Heavy Forwarders (with one used as also a Deployment Server)

My problem is using DB Connect 2.0.5 from the two Search Heads to query an MSSQL database returns "0 rows", while executing the same SQL query from the Search Peers in the indexer cluster returns the expected data from the MSSQL database. The query execution on the Search Head has a green check to show there are NO errors... but unfortunately zero data is shown.
I need to have the DB queries working from the Search Head.

Perhaps the way I installed DB Connect is not correct! Here's how I've tried deploying DB Connect into the Splunk environment...

First, I installed DB Connect directly onto each of the TWO Search Heads. I didn't use the Deployer because the TWO Search Heads are NOT clustered because we need at least three for Search Head Clustering.
Then, I used the Deployment Server on the Heavy Forwarder to push the DB Connect 2.0.5 App into the Master Cluster Node running on one of the Search Heads
Then, I used the Master Cluster Node to deploy the DB Connect Package Bundle into the Indexer Cluster nodes. This returned a great number of VALIDATION errors. I tried skipping the VALIDATION, but that need not get rid of errors when I tried to APPLY the bundle on the Indexer Cluster NODES. Nonetheless the DB Connect Package Bundle seemed to get placed on the two Indexer nodes slave-apps directory. But when I execute an SQL query from the Search Heads using DB CONNECT it complained about missing the DBXquery script on the Clustered Indexer Nodes.

I then removed the DB Connect Package Bundle from the Indexer Cluster Nodes using the Master Cluster Node, and then installed it directly on each of the Indexers in the Cluster. Then SQL Queries then returned 0 rows when executed from the Search Heads, BUT return data as expected when executed from each of the two Indexer NODES in the cluster.

Anybody experienced this situation and know how to remedy it?
Any one who knows the official way to deploy DB Connect 2.0.5 to an indexer cluster with NO search head cluster?
Any one know where I can find documentation on how to install/deploy DB Connect 2.0.5 into an environment where the Indexers are clustered, but the Search Heads are NOT clustered?

Thanks for all your help.

dolivasoh
Contributor

I'm also getting the cannot find dbxquery error after upgrading to 6.3. Wasn't an issue pre 6.3.

I have, in this particular deployment, 1 search head, 2 indexers and a master similar to you but without heavy forwarders. Prior to 6.3 I didn't need the app on the indexers. It seems now that in 6.3 more things are published in the distributed search bundle than before? It's like more search functions have now been moved to the indexing tier (probably to better enable parallel processing pipelines?)

I haven't gotten to forcing the app on my indexers as a workaround like you've done and it seems that won't do me much good anyway.

The way I understand it is the recommended way to use dbconnect is on an ad-hoc searcher, not part of your cluster and not a distributed searcher of your indexers but as a heavy forwarder instead. I don't see many people do this because how are your lookups going to work?

For your solution, I'd recommend configuring dbconnect solely on your heavy forwarders and even have a third instance running the RPC to avoid duplicate queries.

For now, I've gone back to dbconnect v1 where possible on 6.3 instances which seems to work just fine.

0 Karma

napomokoetle
Communicator

Hi dolivasoh,

I applied Splunk DBCONNECT 2.0.6 released yesterday on my Search Head this morning and the problem is now history. Perhaps that will resolve your problem too since it seems to be the same one I had.

https://splunkbase.splunk.com/app/2686/release/2.0.6/agree/

I wish you success.

0 Karma

napomokoetle
Communicator

Hi dolivansoh, thanks for confirming that I'm not going crazy 😉
I was hoping Splunk would have provided me with a tested working procedure by now for the specific setup as the official documentation does NOT contain the particular scenario.

I've been stuck for two weeks now and am beginning to risk falling behind on my deliverables... 😞

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

Hi,

DB Connect is not intended to be put into indexer clusters.

There's two general ways to use the add-on: As a data pump (RDBMS->indexes or indexes->RDBMS or RDBMS->lookups), and as an interactive browser (dbxquery). The first should be done from heavy forwarders, and the second from search heads.

rk60422
Explorer

I am trying to build an input using DBConnect 2.0.4 on Splunk 6.2.3 clustered search heads.

I fill in all the blanks for Identity and Connections. I test the query and it returns results.
I can execute (dbxquery connection=GEIS_SN query="SELECT%20*%20FROM%20vw_Cloud_SN_INC" maxrows=100) and get back data.

When I schedule the job and tell it to write to one of my existing CLUSTERED indexes, the job completes successfully with zero rows indexed.
When I schedule the job and tell it to write to an existing LOCAL indexes, the job completes successfully and the data is available for searching.

How do I get it to write to my clustered indexes?

0 Karma

napomokoetle
Communicator

Hi,
I'm doing the second(dbxquery), and I'm doing it from a search head. However dbxquery doesn't work unless I have DBCONNECT installed on the indexer cluster. It returns errors complaining about dbxquery missing on the indexer cluster. But then when I install DBCONNECT on the cluster, the queries execute without errors but return zero rows!

How then would I command/configure the Search Head to bypass the indexer cluster for DBCONNECT?

Thanks for your help jcoates.

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

you shouldn't have to do anything... the question here is why is it not working when you just have it installed on the search head?

0 Karma

napomokoetle
Communicator

Is it not because when the indexers are clustered, they actually perform the search function, and the search head only combines the results?

Seems to me the indexers are succeeding in executing the queries, but for some odd reason instead of returning the actual records retrieved from the database, return zero rows to the search head.

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

If that's indeed what's happening, it's probably not what you want, because it would result in each node asking the same question of the database at the same time...

0 Karma

napomokoetle
Communicator

Yeah I hear and understand what you're saying. Could it be that the double query from the indexers is then confusing the results?

How then does one configure the Search Head such that it uses DBConnect without involving the indexers? Is there specific documentation that explains how the Search Head must be configured to use DBConnect directly in an environment that has clustered indexers?

Thanks again for your assistance jcoates.

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

everything that we have in docs is here... http://docs.splunk.com/Documentation/DBX/2.0.5/DeployDBX/Distributeddeployment

I'm still not understanding why the custom command would be sent to the indexers though. It's supposed to be local (http://dev.splunk.com/view/python-sdk/SP-CAAAEU2).

-bash-4.1$ cat splunk/etc/apps/splunk_app_db_connect/default/commands.conf 
[dbxquery]
filename = dbxquery.py
supports_getinfo = true
supports_rawargs = true
passauth = true
run_in_preview = false
local = true
-bash-4.1$ 

napomokoetle
Communicator

Hi,

I did follow that document you refer to for installing DBCONNECT on Splunk Search Heads jcoates. Thereafter I and configured as documented and got error stating the dbxquery script is missing on indexers every time I tried to execute a query from the Search Head. This prompted me to deviate from the document and install the DBCONNECT to get rid of the error.
Would you recommend I uninstall the DBConnect app from the Indexers?
If so, should the error around the missing dbxquery.py script on indexers return, how do I then tell the Search Head to stop contacting the Indexer to accomplish the query?

My commands.conf reads precisely like your the one you shared jcoates:

[root@JHBTLSPLxxx ~]# cat /opt/splunk/etc/apps/splunk_app_db_connect/default/commands.conf
[dbxquery]
filename = dbxquery.py
supports_getinfo = true
supports_rawargs = true
passauth = true
run_in_preview = false
local = true

Where JHBTLSPLxxx is my Search Head

jcoates_splunk
Splunk Employee
Splunk Employee

confirmed that this is a bug -- thanks for your help, we'll get it fixed ASAP.

napomokoetle
Communicator

Applied the DBCONNECT 2.0.6 on my Search Head and wallah... Problem solved!
You rock jcoates!
Thank you very much for your responsiveness and effort on resolving this matter.

todd_miller
Communicator

Thanks for the info, jcoates. No more errors regarding the indexers. Now I'm back to my original problem of information to being returned in the query for the lookup.

0 Karma

napomokoetle
Communicator

Thank you very much for the update jcoates and all the attempts to help me resolve the issue. Hope to hear from you soon regarding the fix.

0 Karma

napomokoetle
Communicator

Is there perhaps some work-around in the meantime while you look into a permanent fix for the bug, that would enable me to continue using DB Connect from the Search Head without altering the architecture?

Thanks!

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...