Updating to the latest version of the Splunk Add-on for Salesforce v2.0 has caused what I believe to be a bug, where Splunk misses events for objects that are being monitored. I have reproduced this issue across a production system on Splunk 6.6.2 and a test instance on Splunk 7.7.3. I have not experienced this issue on earlier versions of the Salesforce Add-on. It's difficult to explain via text so I will describe how you can easily reproduce it in the hopes that it is easier to read.
Set up any input to ingest an object in Salesforce with LastModifiedDate, the default Salesforce field, as the object field for "Order By". If you have n number of these objects in Salesforce that have a LastModifiedDate more recent than your query start date (if you left query start date blank the default is 90 days ago) then you would expect the add-on to ingest and index all n objects. However, you'll find that it only indexes n-1 objects. The object with the latest LastModifiedDate will be missing. I'm confident Splunk does see this event though because the SOQL that is sent to Salesforce (identified by the _internal index with INFO logging) returns n number of events which I've confirmed using the Salesforce SOQL developer console. Further evidence supporting that Splunk sees this latest event is that Splunk will update the Query Start to be the date of the most recent object's LastModifiedDate. Therefore the object is seen by Splunk, but just doesn't seem to get indexed.
Furthermore, you can force Splunk to index this or any object by editing two or more objects in Salesforce in-between the time period of the interval. For example if you have the interval set to query Salesforce every 10 minutes and you edit Object A, B and then C in this time period then Splunk will index Object A and B, but there will be no event generated for C. However, Splunk will use the LastModifiedDate of Object C for future queries. You can find further evidence that Splunk is missing an event by searching "index=_internal events collected" with INFO logging configured in the Salesforce app and you will see "file=cce_plugin_sfdc.py, func_name=log... Events Collect n-1".
I suspect this is a simple fix by editing one of the python functions in the app so that Splunk indexes the latest event, but I haven't quite figured out a fix. Has anyone else experienced this issue or know of a workaround?
... View more
