The idea: Perform Splunk development in a non-prod instance as to not impact production searches/alerts/dashboards, and ensure a high quality experience for users in Production.
I believe this is technically possible as I know Splunk Cloud instances which use ES and ITSI usually get placed on a separate search heads pointing to the same indexers. Can I request and/or purchase an additional search instance for to doing app development/QA?
I'd prefer not to have to fiddle with hybrid search/security and maintaining additional Splunk instances outside of Splunk Cloud.
Splunk Cloud Search Head 1 (Dev/QA): I would develop/test new dashboards and/or updates for production here. This search head would point at the production indexers so I can test with production data.
Splunk Cloud Search Head 2 and/or Search Cluster (Prod): Once I tested, Splunk support or I would promote application from Search Head 1 (Dev/QA) to (Prod) Search Head 2 / Search Cluster.
I was able to work with our sales rep. It seems you can purchase additional Splunk Cloud search heads either connected to your existing Splunk Cloud indexers in Splunk Cloud or a standalone(all-in-one) Splunk Cloud instance. They require a minimum additional purchase of 5GB of Splunk Cloud license.
Yes - This is possible within a Splunk Cloud environment. I have multiple search heads for multiple purposes - 1st is a Non-Enterprise Security search head (for non security-related stuff) the 2nd is an Enterprise Security search head (for security-related stuff) and the 3rd is just an "inputs" search head (does the heavy lifting to inform other search heads).
As for indexers, I'm relatively certain that this can be accomplished, however, it's not something that I have done personally.
You should speak to the Splunk Cloud team about your specific circumstances.
@MikeElliott did you request the ES/Non ES instances yourself or was this just the default? Do you know what your allowed search head limit is? I have heard you could can get the separate data input instance with a request to support. I think it's called IDM - Independent Data Management instance (essentially a Heavy Forwarder running things like DB Connect in Splunk Cloud that won't impact your search heads with its workloads)
Thanks for sharing,
Apologies for the looooong delay in replying - I'm not here very often (in case you couldn't tell!).
Unfortunately, I don't have a good answer for you as we're a managed service, so most of this is handled by other teams prior to on-boarding into us.
Best people to speak to are Splunk Support or Cloud Operations.