Using Splunk Cloud and Triggering On Premise Actions

[LCelley]

We're running a Splunk Cloud environment and are trying to figure out how we could trigger an on premise script to restart a service when Splunk sees issues in the logs. From what I've read in documentation we'd be looking to run an adaptive response action, but that seems to be limited to ES customers. We're not currently paying for ES but it lists the action on a general Splunk Cloud page. Hybrid search from an on premise heavy forwarder sounded like another alternative, but it does not allow scheduled searches.

Moreso I'm looking for input on what other Splunk Cloud customers are doing to run scripts on premise from alerts.

From Splunk Cloud service description:
- Splunk Cloud Platform does not provide system-level access. This means you cannot define alerts that run operating-system scripts or use other system services (although vetted and compatible apps can do so). Alerts can be sent by email or HTTPS POST using Splunk software webhooks. You might be required to set up an endpoint inside your network. If you have both Splunk Enterprise and Splunk Cloud Platform, you can run an on-premises search head to support searches that require alert actions. For more information, see Set up an Adaptive Response relay in the Administer Splunk Enterprise Security Manual.

https://docs.splunk.com/Documentation/SplunkCloud/8.1.2101/Service/SplunkCloudservice 

From hybrid search documentation:
- Only ad-hoc searches are supported. Scheduled searches are not supported.

https://docs.splunk.com/Documentation/SplunkCloud/8.1.2101/User/SearchCloudfromEnterprise 

[tscroggins]

@LCelley 

As noted, you'll need to deploy a service exposed to Splunk Cloud email or webhook output. There are countless ways to do this, but the main problem is security. Your options for authenticating and authorizing commands sent from Splunk Cloud are very limited.

An on-premise search head ("hybrid search") is probably a much safer option. I'm not sure why Splunk limits the ScheduledSearch feature in this license context, though. You'll want to contact your sales team (not support) and discuss your use case.