How do I load custom app config (local) to my Splunk Cloud deployment?

[tpavlik_splunk]

I have an app installed -- Splunk_TA_remedy -- and I'd like to change some configuration properties in the alert_actions.conf but I can't see a way to do this in the UI. I'm considering forking Splunk_TA_remedy and packaging these config as a separate app to install onto my deployment to override the config in Splunk_TA_remedy. In my Splunk Enterprise deployment I would simply make these changes within $SPLUNK_HOME/etc/apps/Splunk_TA_remedy/local/alert_actions.conf. How can I achieve the same in Splunk Cloud?

[tpavlik_splunk · Accepted solution]

Firstly, please don’t fork the app. Forking will cause the forked version to lose any support provided to the original app. It will also create a situation where the app doesn’t receive important updates, manual intervention and assisted installs are required thereby dramatically slowing time to value.

A better practice is to use the REST APIs to make edits to the configuration just like Splunk web does during normal usage!

Instructions

In order to make changes to the settings of an app, you’ll need to gain programmatic access to the Splunk instance, fetch the existing settings, post your new settings, and then check your work.

1. Allow Search head API access using the IP allowlist via the Settings > Server Settings menu, see: https://docs.splunk.com/Documentation/SplunkCloud/latest/Admin/ConfigureIPAllowList#Add_or_remove_su...

You can allow the specific IP subnet that you are making cURL commands from, or allow '0.0.0.0/0' to allow API access for all IPs.

2. Create a Splunk Authentication Token to make REST API calls with under the Settings > Tokens menu and ensure that the token is on behalf of a user with sc_admin privileges or higher. You may need to Enable Token Authentication first before proceeding. See: https://docs.splunk.com/Documentation/Splunk/latest/Security/UseAuthTokens

3. Export the Splunk Authentication Token created in step 2 for use with cURL:

$ export TOKEN='eyJraWQiOiJ...'

4. Make a REST call using cURL to GET the properties of a given conf file for a given app/user. The `nobody` user can be used to alter <app>/local configurations across users. See: https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTconf#properties.2F.7Bfile.7D.2F.7Bst...

Format for the URL being requested:

https://${STACK_NAME}.splunkcloud.com:8089/servicesNS/nobody/${APP_NAME}/properties/${CONF_NAME}/${STANZA}

For example here we're retrieving the properties from the alert_actions.conf file and [remedy_incident] stanza within the Splunk_TA_remedy app:

# Note: for readability we use the jq library but the "| jq .entry" can be removed
$ curl -X GET -H "Authorization: Bearer ${TOKEN}" "https://${STACK_NAME}.splunkcloud.com:8089/servicesNS/nobody/Splunk_TA_remedy/properties/alert_actions/remedy_incident_rest?output_mode=json" | jq .entry
...
  {
    "name": "param.impact",
    ...
    "content": "1-Extensive/Widespread"
  },
  {
    "name": "param.urgency",
    ...
    "content": "1-Critical"
  },
...

5. Make a REST call using cURL to POST the properties of a given conf file for a given app/user. The nobody user can be used to alter <app>/local configurations across users. See: https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTconf#properties.2F.7Bfile.7D.2F.7Bst...

Format for the URL being requested:

https://${STACK_NAME}.splunkcloud.com:8089/servicesNS/nobody/${APP_NAME}/properties/${CONF_NAME}/${STANZA}

For example here we're setting the param.impact and param.urgency properties within the alert_actions.conf file and [remedy_incident_rest] stanza within the Splunk_TA_remedy app:

$ curl -X POST -H "Authorization: Bearer ${TOKEN}" "https://${STACK_NAME}.splunkcloud.com:8089/servicesNS/nobody/Splunk_TA_remedy/properties/alert_actions/remedy_incident_rest" -d 'param.impact=4-Minor/Localized' -d 'param.urgency=4-Low'

<?xml version="1.0" encoding="UTF-8"?>
<response>
  <messages>
    <msg type="INFO">Successfully modified 2 key(s)</msg>
  </messages>
</response>

6. Repeat Step 4 to validate that the properties that were changed in Step 5 have new values. In our Example:

# Note: for readability we use the jq library but the "| jq .entry" can be removed
$ curl -X GET -H "Authorization: Bearer ${TOKEN}" "https://${STACK_NAME}.splunkcloud.com:8089/servicesNS/nobody/Splunk_TA_remedy/properties/alert_actions/remedy_incident_rest?output_mode=json" | jq .entry
...
  {
    "name": "param.impact",
    ...
    "content": "4-Minor/Localized"
  },
  {
    "name": "param.urgency",
    ...
    "content": "4-Low"
  },
...

[rrossetti]

Does the POST endpoint work for Splunk Cloud? The documentation linked is for Enterprise. The original question, and mine is regarding Splunk Cloud platform. 

https://${STACK_NAME}.splunkcloud.com:8089/servicesNS/nobody/${APP_NAME}/properties/${CONF_NAME}/${STANZA}

SCP has limitations on what admins can do via the API for modifying app configurations.

https://docs.splunk.com/Documentation/SplunkCloud/9.2.2403/RESTTUT/RESTandCloud