DB connect missing events

[tomapatan]

Hello,

We ingest data from a database using rising columns, however a small amount of events are missing from the index, although I can see them in DBConnect.

The field that we use as a rising column is set as an identity column so I`m expecting that each new value is generated based on the current seed & increment.
Query timeout is set to 30 seconds, max rows to retrieve is 0 (maximum),  fetch size is 300 and frequency is 60 seconds - from what I`ve observed this should be sufficient for our requirements.

Any assistance would be greatly appreciated.

Many thanks.

[tomapatan · Accepted solution]

Update:

DBConnect is sending the logs to both our Cloud and On Prem instances - some events are missing from the Cloud indexer, although they are present on the local indexer.
We`ve raised a support ticket with Splunk to investigate.

[etoombs]

Are there any errors in the dbconnect logs? How are you verifying that you have missing records? Identity columns sometimes skip a number, so just having a small gap doesn't necessarily mean a missing record (databases can skip a number in an identity column if an insert is attempted and fails.)

[tomapatan]

Hello,

Can`t seem to find anything in the _internal index and the DB Connect Health dashboard doesn`t appear to be working. - I`m currently looking into this.

"How are you verifying that you have missing records?"

- I use the "delta" command to compute the difference between the current value of the rising column field and the previous value. The gaps are only present in the Splunk index and I can see all the rows incrementing as expected in the DB connection. 

- I also compared the total number of events with the total number of records in the database over a given period of a day and the results are inconsistent: most days they are correct, but every few days there is a discrepancy.