All Apps and Add-ons

Splunk DB Connect: How to send database search output to Splunk and build a dashboard with the result set?

Path Finder

Hi all,
My requirement is, I have to build a Dashboard by using a database search output.
I have a complex SQL search with 100+ lines logic in it having complex logic in it.
When I tried to run through Splunk which is another network, Faced problems with Java Bridge server (Status as loading or stopped). And the search job status will be displayed as parsing
Due to which other database cron jobs were stopped.
By restarting Search Head I am able to see Java Bridge server status a s running again.

Now is there any possibility to build a dashboard with the output of complex SQL search without running it through Splunk?
We are using Splunk dbconnect app 1.1.4 version.

Thanks 🙂

0 Karma
1 Solution

Builder

@raj_mpl
DB Connect is now available at version 3.1.4
I would suggest you to upgrade the app as this has improved significantly over 1.x that you are using. As long as you are able to get your query output within the stated timeout value you should be fine irrespective of the complex logic.

To the part that - Due to which other database cron jobs were stopped
This shouldn't be a problem by the app. Splunk DB connect is only establishing a connection with your DB and firing the query you are stating. You might want to check your query performance directly from your DB.

And if you still want to ingest by avoiding DB Connect and build a dashboard in Splunk. You can build a stored proc of your query. Run it on a cron schedule and save your outputs to a csv. queryout argument with EXEC SP is one way how you can do that. But I am sure you can explore about it more. And you can then define a file based monitoring input to your splunk.

Let me know if this helps. Thanks !

View solution in original post

0 Karma

Builder

@raj_mpl
DB Connect is now available at version 3.1.4
I would suggest you to upgrade the app as this has improved significantly over 1.x that you are using. As long as you are able to get your query output within the stated timeout value you should be fine irrespective of the complex logic.

To the part that - Due to which other database cron jobs were stopped
This shouldn't be a problem by the app. Splunk DB connect is only establishing a connection with your DB and firing the query you are stating. You might want to check your query performance directly from your DB.

And if you still want to ingest by avoiding DB Connect and build a dashboard in Splunk. You can build a stored proc of your query. Run it on a cron schedule and save your outputs to a csv. queryout argument with EXEC SP is one way how you can do that. But I am sure you can explore about it more. And you can then define a file based monitoring input to your splunk.

Let me know if this helps. Thanks !

View solution in original post

0 Karma

Path Finder

Hi @amitm05 ,

Could you just elaborate the procedure number 3 . And more over I had scheduled some of the database search queries using |dbquery command in splunk . So any kind of database related search query is ultimately processed by splunk only right?
Please correct me in this.

Thanks 🙂

0 Karma

Builder

The searches that you are making with |dbquery arent getting indexed in splunk. You are just firing these queries directly on your DB through splunk. In splunk you can use the splunk commands on the results captured from this dbquery (if this is what you meant by saying that database releated search query is processed by splunk).

0 Karma

Path Finder

Yes @amitm05 , search queries using |dbquery has to be processed by splunk only but the data for processing will be in database .

I am thinking to develop a script for running the query on database and will keep it as scripted input with a interval=20m like that . Will it work ?

0 Karma

Builder

yes, that is one alternative around it.
I still think, trying to upgrade your DB Connect to the latest might also work this out for you. But I'll leave this choice on to you. Thanks !

0 Karma

Path Finder

I go with scripted input as of now . Could you please help me about the configuration files .
I have the Query with me now . If I place that query in a script , after a successful run of that script ,output will be redirected to a separate file .
Now how to keep my config files in forwarder , at a time executing the script as well as reading the output file.

New to scripted inputs. Thanks

0 Karma