Hello Splunk Gurus, I would like to understand if Splunk has solved this problem about auto-scaling Splunk Indexer-Cluster depending upon the incoming data-volume in AWS via tools like K8s or Terr...
See more...
Hello Splunk Gurus, I would like to understand if Splunk has solved this problem about auto-scaling Splunk Indexer-Cluster depending upon the incoming data-volume in AWS via tools like K8s or Terraform or any-other? What the problem statement states is – Spin more indexing nodes as the data volume increases, automatically Provision an AWS instance with Splunk image Mount the data volume Add the indexer into existing cluster as a peer to store and replicate the buckets Remove indexing nodes as data volume decreases, automatically Inform the Cluster Master about scaling down Remove the indexer(s) from the cluster Unmount the data volume and free-up the disk space back to AWS De-commission the AWS instances Making sure the data is fully available and searchable during this process Purpose of this exercise is - To save the AWS cost since its pay-as-you-use model and if, on the day of less incoming data, few of the indexing nodes can be shut-down since they are mostly underutilized on such days due to less search activities and less indexing data. My biggest concern about auto-scaling is - the fact that buckets are replicated randomly on all the indexers of the cluster, and if on a certain day when there is less data incoming, let's say over the weekends, if n indexer nodes can be shut-down to save cost, data is not completely available. And with SF=2, RF=2, if Cluster is recovered to its full-state with n nodes being shut-down, On Monday there will be so many excessive buckets with those node again becoming part of the cluster to handle the working week-day traffic. Answers I seek - I would like to know the insights about this problem-solving in terms of approach and strategy if someone and/or Splunk has solved it with their Splunk Cloud offering. I would also like to understand and have assessment inputs from the community and Splunk Gurus / Architects if its really a worthy problem to solve or if it makes sense at all, it may be an absurd idea and I am fine learning it. Thanks!