Finally figured out the problem. There was an issue with indexes being pointed to the incorrect indexers, so the search head wasn't able to send the results. Once that was fixed, the problem was resolved.
Thank you to all who provided suggestions and aided in our troubleshooting efforts!
I ran into this the other day in a docker demo environment. After bouncing the HWF, input started working again. It's worth a try if you haven't bounced it yet.
Do you mean bouncing the database server where we are pulling data from via db_connect? Or bouncing the search head where db_connect is installed? Or the Splunk Indexer where we are supposed see the indexed queries? Just for clarification.
If DBX is installed on a search head (SH) and not a search head cluster (SHC) and this is where you're running your inputs, then in your case you would need to bounce the search head.
Are you using rising columns?
Also we no errors are reported in the internal index, so we can't track even if it is a permissions issue.
The original query was using a rising column. During troubleshooting, I created other queries that use batch instead. None work.
Also possibly a permissions issue? https://answers.splunk.com/answers/474713/splunk-db-connect-after-creating-a-rising-column-d.html
I thought so too, so I double-checked. Permissions are default of:
Roles Read Write
db_connect_admin √ √
db_connect_user √