Hi,
We have a platform where lot of dashboards are populated using splunk searches via splunk api call.
All the query runs fine and brings results in 30-40 secs however there is one query which brings results in around 5 mins. The query is not complex and runs within seconds on splunk search heads and I also used postman to check how long it takes- it was around seconds on postman as well. I beleive there is no issue at query end.
I checked the splunk logs for that particular SID which took 5 mins to run: I see below ERROR logs:
ERROR SHCRepJob [18757 SHPPushExecutorWorker-0] - failed job=SHPRepJob peer="splunk-search head", guid="xxx" aid=xxx, tgtPeer="splunk-search head ", tgtGuid="xxx", tgtRP=xxx, useSSL=false tgt_hp=xx:8089 tgt_guid=xx replicate?output_mode=json, error=500 - Failed to trigger replication (artifact='xxx Job id') (err='event=SHPSlave::replicateArtifactTo invalid status=alive to be a source for replication')
Any idea about this error. Because i do see this error on other queries SID which are running fine. I am not sure what exactly is the reason for delay.
