All Apps and Add-ons

Splunk Add-On for Salesforce: Why are some rows missing in custom object?

slnsalim
Loves-to-Learn Lots

Hi

I'm using this add-on on SplunkCloud to index custom Salesforce objects and using the LastModifiedDate as the query criteria. When I look at the Salesforce queries in  the _internal logs, I see Splunk is periodically skipping over some rows

For instance, Splunk will send this query to get the next batch of records to index

SELECT .. WHERE LastModifiedDate > 2021-11-25T10:10:51.000+0000 ORDER BY LastModifiedDate ASC LIMIT 1000

 

The FIRST ROW in the result has the LastModifiedDate of 2021-11-26T0910:04.000Z - which I would expect Splunk to use in it's next indexing round. However,  the next entry in the _internal logs sends a different dateTime effectively missing data logged between 9:10:04 and 09:15:33

SELECT ... WHERE LastModifiedDate=2021-11-26T09:15:33.000+0000

 

I'm making an assumption this is how the Add-On works as I can't find any documentation that explains it. Has anyone had this issue and more importantly, found a fix?

I query the _internal logs using this search

index=_internal sourcetype=sfdc:object:log "stanza_name=<my stanza>"

 

Thanks!

 

Labels (1)
0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

@slnsalim , @superdan07 

Are you seeing data also being missed? (What you see on Salesforce vs What you see on Splunk)

If so then that could be an Add-on code issue. I checked the code high-level but it requires more troubleshooting. Please check with Splunk support.

0 Karma

slnsalim
Loves-to-Learn Lots

Hi

Yes, data is missing in Splunk that is in Salesforce. I am unable to raise a case with Splunk as I don't have an active support contract (I've tried). Any other suggestions please?

This add-in would have afforded centralised logging and auto alerts with minimal effort (a win-win) but sadly, in my case at leaset,  I've found it to be unreliable.

Only one other person seems to have experienced the same thing so could it be a configuration issue? frequency of calls to Salesforce? Any best practice tips would be welcome.

Thanks

Sarina

0 Karma

superdan07
Loves-to-Learn

I am having similar issues... did you ever get it resolved?

0 Karma

slnsalim
Loves-to-Learn Lots

Sadly not. I'm having to now look at a PUSH model instead of a PULL which would require bespoke coding in SF. Not my preferred route. I still live in hope though that an answer will be found. If you do find one, would appreciate it if you could share it here. 

 

Thanks

Sarina

0 Karma

superdan07
Loves-to-Learn

@slnsalim @VatsalJagani 

I believe the problem has to do with the Order by LastModifiedDate Splunk uses by default in the input configuration.  When I used this I had some missing rows from the object not sure why but this will vary from object to object and how the data was originally entered in SF. In my case records from the object in  2021, 2022 were accurate but had missing data from 2020 and 2019. I tried using a different Order by field (need to make sure that this is a valid date field with both date and time stamp) and got more results but had a few missing more records. I finally decided to use two inputs for the same object and used two different indexes in Splunk, this caused some duplication when I did my Splunk search with index=SF1 OR index=SF2. I added a dedup command to the search and I got a complete results.  I turned off the other input that was not using the LastModifiedDate field since it gave me the missing records. The input that uses LastModifiedDate seems to be working fine for newer entries and I left it active. Something important to note that the records that I was missing was from years were did not have SF and were backdated when entered in SF. 

0 Karma

slnsalim
Loves-to-Learn Lots

@superdan07 

I don't have another date field apart from CreateDate and that won't give me the latest changes to the record when queried.

How have you configured your 2 inputs? What fields are you ordering by and are you ordering by the same fields in both inputs? What is your polling period set to?

 

Thanks

Sarina

0 Karma

superdan07
Loves-to-Learn

@slnsalim I am polling every 1200 seconds with a limit if 1000. Both inputs have the same Object and Object fields, the only difference is the Order by field. My object is cases, and I have case date as well as LastModifiedDate both of these have a date and time stamp. So input 1 uses LastModifiedDate and input2 uses case date. The dates are usually close to each other don't always match so I am using a dedup on a field that I know should be unique on each record.

Double check in SF and get a list of all date time fields. I believe Chrome web browser has a add-on that allows you to query any object in SF and get a list of all fields.

0 Karma

slnsalim
Loves-to-Learn Lots

Thank you @superdan07 . I'll give it a go.

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

Yeah, that is a good point. Using the right field for checkpointing is very important.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...