Splunk DB Connect App not putting data in Splunk index

[raidercom]

Hi:
I'm trying to get SplunkDB Connect app to pull data from an Oracle database into Splunk.

Working:

Database Connection

DB Info

DB Query with the SQL statement I'm using

Not Working:
When I go to setup the query to actually pull data into a Splunk index, I cannot seem to get anything to go into the index from the query.

What I've done:

Created a new index for the data

Specify the SQL query in a new database input

Scheduled query

New index in the spunk app & new index in the dbx app

Every 1/2 hour & auto

dump (not tailed, wanted to keep it simple to start)

no source type & tried with a new sourcetype

Output formatting:

Key-Value format

Output timestamp with proper field and value & no output timestamp specified

Before restarting Splunk after specifying the new index, I've gotten this message, with different variations on the information contained within based on what I've chosen on the query:
This message has seemed to go away after rebooting Splunk

earch peer has the following message: received event for unconfigured/disabled/deleted index='dbxtest' with source='source::dbmon-dump://' host='host::' sourcetype='sourcetype::dbmon:kv' (3 missing total)

Things I have not tried:

Custom source type, don't see the point

Look-up table, again, don't see the point

Maybe I'm looking at this the wrong way, but I'm trying to test whether there is data in the index by searching for index=dbxtest range:all time from within the splunk dbx app

Thank you for any information/assistance you could provide.

[raidercom]

2013-08-14 14:20:06.159 dbx9265:INFO:DumpDatabaseMonitor - Database monitor=[dbmon-dump://DMART/DBXTest3] finished with status=true resultCount=226687 in duration=6157 ms
2013-08-14 14:20:06.159 dbx9265:INFO:ExecutionContext - Execution finished in duration=6158 ms
2013-08-14 14:20:06.159 monsch1:INFO:Scheduler - Execution of input=[dbmon-dump://DMART/DBXTest3] finished in duration=6157 ms with resultCount=226687 success=true continueMonitoring=true

[raidercom]

2013-08-14 14:20:00.001 dbx9265:INFO:DumpDatabaseMonitor - Executing database monitor=[dbmon-dump://DMART/DBXTest3]
2013-08-14 14:20:00.297 dbx9265:INFO:SpoolOutputChannel - Moving temporary file /opt/splunk/var/run/tmp/dbx/kv_7011579052063595050.dbmonevt with size=5243011 to destination /opt/splunk/var/spool/dbmon/kv_1376504400296939423.dbmonevt

[raidercom]

Nothing strange in the dbx log, just the execution successful messages.
Splunkd.log doesn't have anything related to dbx

The index still appears to be empty

The query works perfectly from the DB Query page, returning the same # of results as the scheduled query.

[lukejadamec]

In the dbx log from the time the input was saved to the entry you posted above, do you see anything strange?
Do you see anything in the splunkd.log?
Is the index still empty?
Does the query work from DB Connect > DB Query? I know that sounds like a dumb question, but just verify it.

[raidercom]

Hmm.. I've created a new index, created a new database input. I get the same thing.

2013-08-14 08:20:07.671 monsch1:INFO:Scheduler - Execution of input=[dbmon-dump://DMART/DBXTest3] finished in duration=7635 ms with resultCount=226669 success=true continueMonitoring=true

Even weirder, have the exact same query, database connections, indexes on a test server, and it works there. Have a problem with the dates on the test server, but that is another story.

[raidercom]

Will do.

I'll test this in the next day or so.
Thank you VERY much for the assistance you've provided.

[lukejadamec]

I know I've seen this before.
Recreate the dump or tail with a new input name and check both splunkd.log and the dbx.log for errors or warnings. I know you said the log said success=true, but something else must have failed.

[raidercom]

Hmm... The index appears to be completely empty.

Manager>Indexes shows size of 0 for the index
There don't appear to be bucket folders in the index folder.

[lukejadamec]

Ok, so it is definitly not a conflicting inputs.conf.
Is the target index for this input only? If so, do you have any data in it? You can check this two ways: Manager>Indexes check the size, or check for bucket folders in splunk/var/lib/splunk/indexname/db/. The bucket folders are named 'earliest epoch time_latest epoch time_incrementing unique bucket number.

[raidercom]

If I'm understanding your comment correctly, inputs.conf from other apps should have input configuration lines relating to the dbx app, and that is causing the conflict.

grep -r * from $SPLUNK/etc/apps

only returns an inputs.conf in the dbx folder.

I did the same setup on a test Splunk server (Windows, not Linux) with the exact same settings, and it is pulling data.

[lukejadamec]

raidercom,
Wanna bet?

[raidercom]

The inputs.conf files are pushed by puppet to the splunk server, and the dbx inputs.conf is not. None of the other inputs.conf are able to contain something related to the dbx files unless it is added to the puppet template. The only inputs.conf that has dbx related stuff in it is the one in the dbx directory\apps\dbx\local\inputs.conf
-Jeff

[lukejadamec]

No doubt dbx can be frustrating.
When I said conflicting inputs.conf I meant in different app directories. If you go to Manager > Inputs from the Launcher screen and save an input for dbx it will save the input stanza in launcher/local/inputs.conf, if you get there from search, it will save it in search/local/inputs.conf. I had many copies of my input stanzas when I started with dbx.

[raidercom]

Tested from 2007->2014 search for the specified index. Finds nothing.

I tried with another, new index, same thing. I have to admit that I do not get this...

there is only 1 inputs.conf in the dbx app, and all of the references to indexes in it are pointing to a single index, the correct index.

[lukejadamec]

Your data is there, 226391 records of it. The question is where.
Verify your index is correct, and that you don't have a conflicting inputs.conf.
Search from now to a month in the future (I don't think all time does the future).

[raidercom]

I am seeing that entry in the log:

2013-08-08 10:30:05.568 dbx2490:INFO:DumpDatabaseMonitor - Database monitor=[dbmon-dump://] finished with status=true resultCount=226391 in duration=5567 ms
2013-08-08 10:30:05.568 monsch1:INFO:Scheduler - Execution of input=[dbmon-dump://] finished in duration=5567 ms with resultCount=226391 success=true continueMonitoring=true

When I go to Splunk Web, and go to app\Splunk DB Connect, then go to search, and search for
index= for all time, Splunk shows nothing. Am I doing my query wrong then?
Thanks
-Jeff

[lukejadamec]

Does the dbx.log do you see something like:
dbmon-tail://database/input, resultCount=integer, success=true ?
If so then the data is in your index - try a search for all time.
If you don't see that entry in your log, then what are you seeing?

[raidercom]

From the database input tab.

[lukejadamec]

When you say the 'query' do you mean from the query tab, or do you mean the database input.
Dump is easier, maybe a little too easy.

[raidercom]

Hmm... The query seems to be running successfully (the Oracle date conversion you provided removed the invalid date that Splunk was trying to use), but same thing, no output in the Splunk index. Is there a reason you are suggesting to use tail rather than a full dump? I figured for testing a dump would be easiest.

[raidercom]

OK, just testing the changes.
Thank you VERY much for your assistance!
-Jeff