DB Connect Checkpoint Value Not Updating

[bschaap]

I configured a DB Connect rising input which queries SQL Server but the checkpoint value is not updating which causes duplicate data to be ingested. I have many other DB Connect inputs which update the checkpoint value successfully. The input that doesn't work uses a query with a large number of columns and the data returned can have as much as 4 KB in a single field but average about 1.4 KB per event across all events. Usually a few hundred to a few thousand events are collected each time the input runs. Most of my other inputs don't have as many fields and the max data in a single field is smaller. I can't think of any other differences between inputs. The input was initially configured to run every minute with a timeout of 30 seconds. I was receiving query timeouts so I increased it to 600 seconds and run every 10 minutes. DB Connect is running on a Heavy Forwarder which forwards data to Splunk Cloud.

Has anyone else encountered this problem? What troubleshooting steps have you taken?

[ejenson_splunk · Accepted solution]

I had the exact same issue you had or are having. I found that the next event to be ingested had a bad date prior to 1/1/1970 and that field is mapped to the _time field in Splunk. I updated the appropriate input in /var/lib/splunk/modinputs/server/splunk_app_db_connect to force the input to skip that record, restarted splunk and everything started working again.

[IgorB]

Apparently there are numerous causes for such problem - mine was that some of the data returned by SQL query was causing HEC issues, so the the input was advancing till it encountered a problematic event and then was indexing same set of entries over and over again because behind-the-scenes HEC was failing mid-flight, so checkpoint never got updated.

[bschaap]

I was encountering HEC errors as well. If I recall one showed listed http 503 and another listed java socket exceptions. Increasing maxKBps in limits.conf addressed these errors. Another symptom of my issues were that various Splunk queues on my Heavy Forwarders were getting backed up and flagged as being "blocked".

[bschaap]

Thanks. I ended up coming across a few reasons for my issues. Some of the values populating _time were coming across as 1/1/1900 which is a problem with the source system. To address this I modified the query to change the value to something that wouldn't fail. I have reported this bug to Splunk and they are researching.

There were other issues I came across such the maxKBps setting in limits.conf having a value of 256 on the Heavy Forwarder. This setting may be appropriate for a Universal Forwarder but not for this Heavy Forwarder.