Has anyone here set an alert in Splunk monitoring of Salesforce that runs a script on alert action that calls back in to the org? I'm just now looking at the possibility of creating some "self healing" alerting for a recurring issue we have in SF prod (scheduled batch jobs falling out of queue) and saw the option to run a script in Splunk on alert. Salesforce has an executeanonymous (string apexcode) method through Tooling API exposed in both SOAP and REST. My thought was to have a class built in my SF Prod org that Splunk could just call in to which would handle the logic of finding what's fallen out of spool & kick off the script to put it back in the hopper.  ... View more

I'm not sure if I'm missing something simple or not, but I've got event logs from my Salesforce instance fed in, as well as the User object, and for some reason I can aggregate on some fields of User but not others ... even though the fields exist in Splunk.   index=sfdc sourcetype=LightningPageViewCSV |join USER_ID [ search sourcetype=sfdc:user | eval USER_ID=substr(Id,1,len(Id)-3) ] |stats avg(EFFECTIVE_PAGE_TIME) by Name   // this works to aggregate by the user's name. Not really useful but it was a test to make sure something came through. The substring is b/c one object uses the 18-char Salesforce Id, the other uses the shortened 15-char Id.    index=sfdc sourcetype=LightningPageViewCSV |join USER_ID [ search sourcetype=sfdc:user | eval USER_ID=substr(Id,1,len(Id)-3) ] |stats avg(EFFECTIVE_PAGE_TIME) by State__c,Loc__c   //no results from this for some reason ... State__c and Loc__c are custom fields on User.   index=sfdc sourcetype=sfdc:user index=sfdc sourcetype=sfdc:user Name="[one of the names from the first query]"   //I run these just to see what I've got in my user object and I can see several people with non-null State__c and Loc__c This is a new dev org I just spun up so I'm not sure if I missed a step in adding these sources or not. The LightningPageViewCSV is an imported static CSV file of the EventLogFile for testing. The sfdc:user was a one time read in of the User object. Both of these are tied to the sfdc index. ... View more

I feel like this is a known issue & I feel like it's been around for a while, reaching out to see if anyone has worked around this. I found one single post related to this where the only suggestion was to change how frequently Splunk reads in data, but tbh not much of an option here because Splunk is already known to peg out DBCPU time in my org. Short version of it is I'm having trouble with Splunk only ingesting a job from the AsyncApexJob object in my Salesforce org once, even though that job will get updated repeatedly as it goes through statuses of Queued, Running, Completed, etc. It's not every job that does this, but it's frequently enough that I can't build an accurate alert off it. There's a release note for add-on 4.2.2 that says this is a known issue: https://docs.splunk.com/Documentation/AddOns/released/Salesforce/Releasenotes#Known_issues however I'm on 4.0.3 of the Salesforce addon and my Splunk Enterprise is 7.3.3.  Has anyone else noticed this, worked around it without changing the frequency Splunk hits the org, is it to be fixed in a future update, etc.? ... View more

I've been having a hard time trying to get a Splunk search that will give me a count of all records in my Lead object in Salesforce where OwnerId = Id of the queue I'm using to manage intake and created date = Today, but every time I search our index I'm getting way more records than I should (last check was 513 in Splunk and 413 in SFDC Production). My query is below. Just to explain the current state of the query and what I've been trying, I've created a CleanNow that's just today's date in Year-Month-Day and a CleanCreatedDate converting the Salesforce CreatedDate to the same Year-Month-Day, and my last attempt to limit the search scope was a subselect to find Ids for records where the owner is not my queue and dump them. The added date columns on the table are just to "idiot check" and try to find why I'm getting a delta. index=sfdc sourcetype=sfdc:lead OwnerId="[Id of my queue]" | eval CleanNow=strftime(now(), "%Y-%m-%d") | eval CleanCreatedDate=strftime(strptime(CreatedDate,"%Y-%m-%d"),"%Y-%m-%d") | where CleanCreatedDate=CleanNow | search NOT [search sourcetype=sfdc:lead OwnerId!="[Id of my queue]" | fields Id] | table Id, Status, OwnerId, CleanNow, CleanCreatedDate, CreatedDate, LastModifiedDate Looking at what the Splunk query gives me and searching Prod with those Ids, I can see Splunk is returning things to me as New / owned by my queue that in Prod are actually Converted / assigned to a human. And not even ones that were just now converted, I mean ones that were converted hours ago. I took one of those Ids returned by Splunk as New / owned by the queue but Prod said was converted hours ago, and did a search  index=sfdc sourcetype=sfdc:lead Id="[Id of the delta record]" And I get three events. 1. Status is New and is owned by my queue (this is what I actually want to see in the return) 2. Status is New and it's owned by a human being (do not want to see) 3. Status is Converted and it's owned by a human being (do not want to see) That Id only appears once in the results from Splunk, and the fact that we had two other entries that don't match what I consider a "hit" means I should not have seen it at all, including the one time it logged as New / owned by the queue. Not entirely sure I'm explaining it right, but basically I need to find a way to recursively search the results to find any Ids that match the original query, but are in there a second (or third, or fourth...) time with a different owner, and drop them from the return. ... View more

Full disclosure, I'm a Salesforce guy rather than a Splunk guy. I'm working with my internal Splunk team to try and ingest my CronTrigger and CronJobDetail objects from my org so I can monitor for some job hang statuses, or when developers make jobs with hard end dates. The Splunk team is getting a 400 error on these objects, and their queries look okay to me. I did find a separate article in here about setting intervals for date predicates (https://community.splunk.com/t5/All-Apps-and-Add-ons/Salesforce-object-response-status-400/m-p/444002) and I've passed that on to them to investigate. I've also suggested they try ingesting a single field from each object and see if they can get anything back. In the meantime, has anybody here ingested these two objects into Splunk? Most of what I'm finding on Google is that a 400 message is a bad query, but can't this error also be thrown if the integration user doesn't have object access? If so I may be at an impasse since these are system objects, not standard or custom. Salesforce has these locked down so tight that even with my level of access I can't view the basic system properties to these, not that I'd be interested in messing with access around them. ... View more