Splunk DB Connect: Why does DB Connect no longer stores results after query?

[datorres]

Issue

DB Connect no longer stores results after query. It was working fine in the past, but now just doesn't work.

Troubleshooting Steps

• Changed query in DB Input to something simpler
• Deleted and recreated DB Input, using as many default values as possible
• Created additional DB Inputs for same database
• Created additional DB Inputs for different databases on different hosts
• Ran tcpdump on all devices to ensure that DB Connect was able to successfully connect, query, and receive results
• Tried using different indexes and sourcetype names for each query
• Tried upgrading DB Connect app from 3.1.0 to 3.1.2.

Results

• Splunk search-head successfully connects to port 3306 on target machine, performs query, receives results.
• Try to query results in Search (timeframe "All Time"); no results.

Additional Info

• Splunk Enterprise v7.0
• Splunk DB Connect v3.1.2

[datorres]

Finally figured out the problem. There was an issue with indexes being pointed to the incorrect indexers, so the search head wasn't able to send the results. Once that was fixed, the problem was resolved.

Thank you to all who provided suggestions and aided in our troubleshooting efforts!

[tmuth_splunk]

I ran into this the other day in a docker demo environment. After bouncing the HWF, input started working again. It's worth a try if you haven't bounced it yet.

[baegoon]

Do you mean bouncing the database server where we are pulling data from via db_connect? Or bouncing the search head where db_connect is installed? Or the Splunk Indexer where we are supposed see the indexed queries? Just for clarification.

[tmuth_splunk]

If DBX is installed on a search head (SH) and not a search head cluster (SHC) and this is where you're running your inputs, then in your case you would need to bounce the search head.

[damiensurat]

Are you using rising columns?

[baegoon]

Also we no errors are reported in the internal index, so we can't track even if it is a permissions issue.

[datorres]

The original query was using a rising column. During troubleshooting, I created other queries that use batch instead. None work.

[damiensurat]

Also possibly a permissions issue? https://answers.splunk.com/answers/474713/splunk-db-connect-after-creating-a-rising-column-d.html

[datorres]

I thought so too, so I double-checked. Permissions are default of:

Roles               Read    Write
db_connect_admin       √        √
db_connect_user        √