All Apps and Add-ons

Regarding DB connect

raghu0463
Explorer

i have scheduled a job in db connect to make a search and push that results to sql server.
but the job is not running automatically, i mean its not pushing any results instead of the search having some results, when im manually running that search in db connect for the first time in the day its showing 0 results but if i run that search for 3-4 times its showing results then im pushing that results to GAR.

0 Karma
1 Solution

Richfez
SplunkTrust
SplunkTrust

DB Outputs are what I expect you are doing.

The biggest trick to doing this right is to align your time periods. Inside the search that you are running you have to specify a timeframe to run over, like, say, "earliest=-1h". Then in the scheduled output frequency you must ALSO set the frequency to be that same period, so in that case set the execution frequcney to "3600" seconds. In your case I'm not sure if this is one of the problems or not, but it's worth double-checking.

In your specific case, you say that the search isn't generating results even if you run it manually. That's a big tip off that there's something wrong with your search, or that you misunderstand your data and there's none of the data you want in existence. In either case, this is a search issue, not a DB Output issue - copy your search into a regular search window and start figuring out why the search does what it does.

We can help with this too, but you'll probably have to give more information about the search, what you expect to see, what it does instead, why you think what it's doing is wrong, and possibly some information on the events that underlie the search.

View solution in original post

Richfez
SplunkTrust
SplunkTrust

DB Outputs are what I expect you are doing.

The biggest trick to doing this right is to align your time periods. Inside the search that you are running you have to specify a timeframe to run over, like, say, "earliest=-1h". Then in the scheduled output frequency you must ALSO set the frequency to be that same period, so in that case set the execution frequcney to "3600" seconds. In your case I'm not sure if this is one of the problems or not, but it's worth double-checking.

In your specific case, you say that the search isn't generating results even if you run it manually. That's a big tip off that there's something wrong with your search, or that you misunderstand your data and there's none of the data you want in existence. In either case, this is a search issue, not a DB Output issue - copy your search into a regular search window and start figuring out why the search does what it does.

We can help with this too, but you'll probably have to give more information about the search, what you expect to see, what it does instead, why you think what it's doing is wrong, and possibly some information on the events that underlie the search.

raghu0463
Explorer

1st : What is meant by set frequency, i didn't use this option till now.

2nd : When i know that there are some results for that search( i came to know that by checking in my original database ) but when i check in the db connect for the same search its not showing any results for the first time. if im trying to run that search for like 6-7 times then its giving the results, here im not modifying any thing in the search not even the time , im just hitting the search option repeatedly ( not continuously waiting for the search to run then when i see the results 0 then im hitting it again )

0 Karma

Richfez
SplunkTrust
SplunkTrust

The frequency is the last step in the DB output wizard. It's this step in the lovely DB Output docs.

Let's make sure we're all on the same page and be sure we're using the same terms.

Database: Use this for the data repository on the SQL side of things, nothing really to do with Splunk.
Query: A SQL statement, does something with a Database.
Search: A search inside Splunk, retrieves data from Splunk indexes (or does an inputlookup, etc...)

Now, let's review what I/we know:

You have a Splunk search. It returns some data. That data then needs to be sent into a database.

So what is it we need to be especially sure to be doing "correctly" in the above use case?

1) The search needs to define the timespan it runs over. Methods for this vary, but generally will involve some earliest=X and possibly some latest=Y statements in the search themselves.
2) The search needs to be put into a db connect output, along with specifying the destination server, DB and so on. (Fields will need to line up right, but the errors involving this step are usually pretty obvious)
3) The execution frequency of the search needs to be set (as per the above link) to match the search timespan you set in step 1.

Numbers 1 and 3 above need to match. So if your search sets earliest=-1d@d latest=@d, then you'll want the execution frequency set to a cron, something like 15 1 * * *, which will then launch the DB output process once per day, at 15 minutes past the first hour of the day (1 AM), every day.

To troubleshoot data going weird, there's a couple of things to check.

First, determine if your search runs correctly. You can copy and paste it from the DB Output edit wizard, or can open the .conf file on your filesystem and copy it from there. Run that search, see what returns. Since it'll have a hard-coded time period in it, the search may return different results from what you may expect. Manually expand/change those time frames until you confirm that it does indeed display the right data. Run it at different times to confirm that the NEXT time it runs, it also returns the right data.

Second, find out what data actually ends up in the DB, if any. You will do this with whatever database management product you have that can talk and run queries against your database. This is mostly outside the scope of a Splunk Answers posting, but if you have problems we can probably still help a bit. Usually involves "SELECT * FROM ORDER BY DESC" and a lot of staring and scrolling around, maybe copy and pasting contents into Excel or Notepad++.

Compare the stuff in the DB with the stuff from Splunk. Analysis may take some time to determine exactly what's there and what's not there. Remember, you can use absolute time frames in both your Splunk search and your SQL query to selectively look at little chunks at a time.

Now that we know what is and isn't making it into the DB, there's a high likelihood that the answer to "how do I fix this" will become apparent.

For what it's worth, I don't know if any db output data shows up in the DB Connect app's "Health" pages. It may, it may not, I'm not sure. I'll have to look later.

Hope this helps!

raghu0463
Explorer

thanks for the reply.

0 Karma

woodcock
Esteemed Legend

I do not understand what you mean by "pushing results". What is "GAR"?

0 Karma

raghu0463
Explorer

when i run search in db connect i will be getting some results, which i want to load in SQL server (GAR- what we call it as) .

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...