Hi, we have a search head cluster where a couple of the search heads where removed by shutting down the VMs. In other words, the search heads wasn't removed gracefully as they should be. Now the remaining search heads is complaining because mongodb can't reach the removed search heads. I'm getting the following error messages: 2017-03-23T12:27:42.296Z I NETWORK [ReplExecNetThread-1919] getaddrinfo("prod-searchhead-x") failed: Name or service not known 2017-03-23T12:27:42.290Z I REPL [ReplicationExecutor] Error in heartbeat request to prod-searchhead-x:8191; Location18915 Failed attempt to connect to prod-searchhead-x:8191; couldn't initialize connection to host prod-searchhead-x, address is invalid Anyone knows how to forefully remove a host from mongodb in the search head cluster, so that we'll get rid of these error messages? ... View more

Hi, We had some search heads in a cluster, running on VMs. A couple of the VMs were deleted without properly removing the search heads from the search head cluster first. Now, Splunk is complaining that it can't reach some of the search heads in the cluster. When we try to remove them from the cluster, Splunk won't do it since it gets no reply from the search heads (obviously, since the VMs are deleted). How to we "force" the search heads out of the search head cluster when the search heads no longer exist? Cheers. ... View more

Hi, I've looked around in the documentation for how to set an archiving policy in Splunk Cloud, but I haven't found an answer yet. I know that the document "Safeguarding Customer Data in Splunk Cloud" talk about archiving, but in this paper they call it archiving when data is rolled from hot to warm. I want to to know if there's a way to roll to frozen in Splunk Cloud, and if so, what's the price of storing data this way compared to storing data in warm buckets? ... View more

Hi, I'm wondering how load balancing in Splunk Cloud work. When i install the splunkcloud.uf app on a local forwarder, the outputs.conf that is created in the app looks like so: [tcpout:splunkcloud] compressed = false disabled = false server = input-prd-p-<id>.cloud.splunk.com:9997 sslCommonNameToCheck = input-prd-p-<id>.cloud.splunk.com sslCertPath = $SPLUNK_HOME/etc/apps/splunkclouduf/default/client.pem sslPassword = <password> sslRootCAPath = $SPLUNK_HOME/etc/apps/splunkclouduf/default/cacert.pem sslVerifyServerCert = true useACK = true Notice that there is only one server listed. When I search for "splunk_server" in my Splunk Cloud it clearly says I have five indexers. Why aren't all those listed behind "server" as normal when using load balancing? I know there is something called the indexer discovery feature, but then I guess I would see a stanza for that in my outputs.conf. Could someone explain this to me? ... View more

Hi, On Splunk Cloud, the admin role has by default access to all non-internal indexes. At a customer's site, we want to retain the the access rights of admins by keeping "available indexes" (with the exception of internal indexes) in their own "access roles." Thus, our admin role would only grant the user the capabilities of the admin, but not any non-internal indexes. Those indexes would have to be granted by their own 1:1 access roles. Our problem is that when we grant a user the custom admin role we've created, that user can't see the graphs in the "distributed management console." It seems like the users aren't allowed to see the _internal index, even though it clearly says in the custom admin role we've created that all internal indexes should be searchable. We've even added the index _internal specifically, in addition to "all internal indexes," just to be sure. What's interesting is that if we let our custom admin role inherit from the default admin role, then it works. Any idea of what could be the problem here? Could there be some sort of "hidden" access right or capability on the default admin role in Splunk Cloud? ... View more