I wanted to bring this issue to your attention. We upgraded from 3.10.0 or DB Connect to 3.11.0 back on November 2022. We use an external HEC destination for DB Connect to send its to before it gets to Splunk instead of the local/built-in DB Connect destination (and have been for over a year). There seems to be a bug sending to an external HEC destination. We started getting complaints in early January 2023 from users that data was missing in Splunk. We temp moved these inputs back to the internal HEC and the issue went away. I setup a test DB Connect on 3.11.0 and setup the same inputs on it but sending to external HEC and then to a test index. We did a search to compare the test data with production data and we saw that throughout the day, there were many times when the inputs ran that the data was not making it into Splunk. The first clue there was an issue was seeing this in the logs every time the inputs ran:
[Scheduled-Job-Executor-3] ERROR c.s.d.s.d.r.HttpEventCollectorLoadBalancer - failed to post events:
I remembered that we upgraded DB Connect back in November so I decided to downgrade back to 3.10.0 on the test DB connect server. The failed to post events error went away and all the data in test and prod matched up with no loss of data.I don't know what changed in DB Connect 3.11.0 and higher (3.11.1 has same issue) but this is a fairly big one for me. I will stay with 3.10.0 for now but someone from Splunk needs to look into this issue.
