All Apps and Add-ons

Splunk Add-on for Salesforce Ingestion Problem

lawrence_magpoc
Explorer

I am able to collect data from Salesforce using Splunk Add-on for Salesforce. However:
Let's say the data I am collecting has the fields "CreatedDate", "CompletedDate", and "Status" which can either be "Processing" or "Completed", if an item of status "Processing" had been ingested to Splunk, when the status of that item is updated to "Completed" in Salesforce, for some reason, Splunk doesn't ingest it. I need the events for both processing and completed as I have to compute the time it took for a job item to be completed. Is there a way solve this problem?

0 Karma
1 Solution

jadengoho
Builder

This may not be the best way to solve this:
I encounter this issue on ingesting ApexJobs , apparently this logs changed every second and Splunk along with Salesforce Add-on doesn't re-ingest data after it updates.
For example:

1/11/2020 7:04:03.000 AM   - ID:00000001 Status="Processing"
1/11/2020 7:04:05.000 AM   - ID:00000001 Status="Completed"

Splunk will ingest the "Processing" but not the "Complete". In my observation it will only re-ingest the data if the update is greater than one minute.

Solution:

  1. Lower the interval time to 20-60 seconds in Salesforce-addon.
  2. Create a script that will ingest the APEXJOBS. If you can find a way to ingest all data along with its ID and Fields using wildcard(*).
  3. Since "Processing" status are ingested, i need to find a way to run the script and get the latest data. But according to this document https://developer.salesforce.com/docs/api-explorer/sobject/AsyncApexJob I need to specify the ID to get a specific information.

In a search i get all Processing Status and run a REST API query. Curl application helps me to accomplish that, and dump the data in the same index but different sourcetype. And i save this query as a savesearch to recapture information that are not being ingested.

Again, this might not be the best solution out there, but this workaround help me.

View solution in original post

0 Karma

pensword
Engager

The data is not being pulled into Splunk when Status changed, right?

Take a look at the ORDER BY clause (SOQL). I wonder if ORDER BY Status might work for you, at least in terms of catching the change. ORDER BY CompletionDate might also work. You can find the query in the api conf files and can also see it in add-on logs. You should also be able to see the SOQL query in the add-on logs.,

0 Karma

jadengoho
Builder

This may not be the best way to solve this:
I encounter this issue on ingesting ApexJobs , apparently this logs changed every second and Splunk along with Salesforce Add-on doesn't re-ingest data after it updates.
For example:

1/11/2020 7:04:03.000 AM   - ID:00000001 Status="Processing"
1/11/2020 7:04:05.000 AM   - ID:00000001 Status="Completed"

Splunk will ingest the "Processing" but not the "Complete". In my observation it will only re-ingest the data if the update is greater than one minute.

Solution:

  1. Lower the interval time to 20-60 seconds in Salesforce-addon.
  2. Create a script that will ingest the APEXJOBS. If you can find a way to ingest all data along with its ID and Fields using wildcard(*).
  3. Since "Processing" status are ingested, i need to find a way to run the script and get the latest data. But according to this document https://developer.salesforce.com/docs/api-explorer/sobject/AsyncApexJob I need to specify the ID to get a specific information.

In a search i get all Processing Status and run a REST API query. Curl application helps me to accomplish that, and dump the data in the same index but different sourcetype. And i save this query as a savesearch to recapture information that are not being ingested.

Again, this might not be the best solution out there, but this workaround help me.

0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Get the T-shirt to Prove You Survived Splunk University Bootcamp

As if Splunk University, in Las Vegas, in-person, with three days of bootcamps and labs weren’t enough, now ...

Wondering How to Build Resiliency in the Cloud?

IT leaders are choosing Splunk Cloud as an ideal cloud transformation platform to drive business resilience,  ...