All Apps and Add-ons

DB Connect Checkpoint Value Not Updating

bschaap
Path Finder

I configured a DB Connect rising input which queries SQL Server but the checkpoint value is not updating which causes duplicate data to be ingested. I have many other DB Connect inputs which update the checkpoint value successfully. The input that doesn't work uses a query with a large number of columns and the data returned can have as much as 4 KB in a single field but average about 1.4 KB per event across all events. Usually a few hundred to a few thousand events are collected each time the input runs. Most of my other inputs don't have as many fields and the max data in a single field is smaller. I can't think of any other differences between inputs. The input was initially configured to run every minute with a timeout of 30 seconds. I was receiving query timeouts so I increased it to 600 seconds and run every 10 minutes. DB Connect is running on a Heavy Forwarder which forwards data to Splunk Cloud.

Has anyone else encountered this problem? What troubleshooting steps have you taken?

Tags (1)
1 Solution

ejenson_splunk
Splunk Employee
Splunk Employee

I had the exact same issue you had or are having. I found that the next event to be ingested had a bad date prior to 1/1/1970 and that field is mapped to the _time field in Splunk. I updated the appropriate input in /var/lib/splunk/modinputs/server/splunk_app_db_connect to force the input to skip that record, restarted splunk and everything started working again.

View solution in original post

ejenson_splunk
Splunk Employee
Splunk Employee

I had the exact same issue you had or are having. I found that the next event to be ingested had a bad date prior to 1/1/1970 and that field is mapped to the _time field in Splunk. I updated the appropriate input in /var/lib/splunk/modinputs/server/splunk_app_db_connect to force the input to skip that record, restarted splunk and everything started working again.

IgorB
Path Finder

Apparently there are numerous causes for such problem - mine was that some of the data returned by SQL query was causing HEC issues, so the the input was advancing till it encountered a problematic event and then was indexing same set of entries over and over again because behind-the-scenes HEC was failing mid-flight, so checkpoint never got updated.

0 Karma

bschaap
Path Finder

I was encountering HEC errors as well. If I recall one showed listed http 503 and another listed java socket exceptions. Increasing maxKBps in limits.conf addressed these errors. Another symptom of my issues were that various Splunk queues on my Heavy Forwarders were getting backed up and flagged as being "blocked".

0 Karma

bschaap
Path Finder

Thanks. I ended up coming across a few reasons for my issues. Some of the values populating _time were coming across as 1/1/1900 which is a problem with the source system. To address this I modified the query to change the value to something that wouldn't fail. I have reported this bug to Splunk and they are researching.

There were other issues I came across such the maxKBps setting in limits.conf having a value of 256 on the Heavy Forwarder. This setting may be appropriate for a Universal Forwarder but not for this Heavy Forwarder.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...