This is really a great question, with many facets to contemplate!   First I'll say, that preference is to go with AWS or GCP and use Smartstore based architectures until we release blob-storage compatible object storage compatibility.   With that being said, attached storage isn't cheap in Azure/AWS/GCP etc. So as you build out your indexing fleet typically we want to consider this.. So hot/warm, cold, retention period, replication factor and search factor. These all will play a part in that architecture decision. From the technical point of view, 8 vcores is well be low our recommended spec for a deployment at your size. And I would most definitely not deploy that ( 8physical/16vcores yes no problem..) If you follow our Azure Best Practices, we do recommend Standard_DS15_v2 instances for larger deployments. ( The small instances we do recommend the lower 8 core boxes.. again do avoid at your scale!) If the cost wasn't an issue, I would look at deploying fewer larger boxes and storage. Depending on search and indexing requirements, you might be able to get from 20 smaller boxes down to 8 to 10 beefier ones.  And assuming even data distribution and performance disks, search and indexing shouldn't be a concern. * If you're on AWS, you can go smartstore and I3en instances that have a larger compute footprint and not be too concerned about storage as we offload to s3. Any additional compute or cache storage can be added horizontally at the indexing tier. This is a great conversation that your account team should be able to faciliate with one of the platform architects at Splunk. You should definitely engage to go through this and come up with a best solution as what you may have now has a lot of questions..     * Not knowing indexing or search workloads, the above is just purely a high level design conversation.    ... View more

There's a few what ifs here.. Are you moving peers one/two at a time or lift and shift everything at once.   The lift and shift is the easiest in regards to moving subnets. Gracefully shutdown all your Splunk instances, move them to the new network, configure the OS, reconfigure Splunk via config files (USE DNS THIS TIME), test connectivity, bring everything back up... Shouldn't be any issue.   If you have to move servers one/two at a time, then it can be a bit more difficult as you need to validate IP and port connectivity across you network (never assume its going to work :>).  Before moving anything, I would go through your Splunk config files and find all references to IP and change them to fqdn's. Then make sure you have DNS correct, and if you can't update DNS, configure your /etc/hosts file to be a substitute for these hosts until you can. From there, you can move your hosts incrementally.    Otherwise your process is accurate and will work.  ... View more

Rsync will work fine for this. Be cautious around the GUIDs of the SH, if you bring production back up while the DR is running you can have some potential issues. You could change this easily enough and not sync this config file. If you're syncing KOs, why no use GIT or similar repo control. This is what most companies are doing now these days. It's a lot easier for granular control of what you replicate across, not to mention the benefits of version control/tracking.  ... View more

Regarding serverless "push" to Splunk, as opposed to Splunk polling. This is the future of real time analytics, and polling will eventually die.  With that, Splunk has support for HTTP Event Collection (HEC), which is a standard logging library in Java and most major languages now. This allows logging to send, via a key, to a remote https location its logs in a structured format.  There's quite a bit of information on this here : https://dev.splunk.com/enterprise/docs/dataapps/httpeventcollector/. Splunk Architecture courses will touch on this a bit, HEC is typically enabled on the indexer(s), and in larger environments a load balancer is put in front of this for clients to send to and distribute events evenly. See more here : https://docs.splunk.com/Documentation/Splunk/8.0.4/Data/UsetheHTTPEventCollector As for adding REST endpoints, this is always a good idea to enable other apps access to your systems, as RESTful connectivity is pretty much standard for webapps now. And with this, as long as you have a REST logging endpoint, Splunk can poll this. This circles back around to having a requirement for a Agent that does polling, and the lack of data during the polling intervals...       ... View more

This is indexed data by Splunk. You can read about what this actually includes in Splunk Docs, more specifically here : Splunk Indexed Data  So this is a local, non-clustered, warm bucket. The 183236610 represents the newest EPOCH time of the data in the bucket and 1832273414 represents the oldest EPOCH time of the data in the bucket. The 19315 is a local ID of the bucket that Splunk uses to keep track of.  ... View more

As you mention, there are a lot of moving parts to this to consider. In general our AWS sizing guidance holds up. I would use the i3en instances over i3 at this time. The performance is better, and this is what Splunk Cloud currently deploys. Additionally, our sizing is more around concurrent searches and not concurrent users, event though they are connected. ( 1 user vs 1 user running 15 searches..) For ES,  we still will say target 100 to 150gb/day per indexer. But that being said, on Splunk versions 7.2+, you should be able to tune and see numbers closer to 200 or even higher (depends heavily on your search workload, DMA etc.)  With that being said and not knowing what your data sources and volumes are, and using 150 as the base line, 20 indexers should be the baseline target for deployment. That isn't a huge variance from your 17 number, that would be about 175gb/day per indexer. And again with tuning, that in general shouldnt be difficult to achieve.   Regarding your search head, I'd look at the M5 series over the M4.. At this scale, I'm curious why you are not deploying a SHC. You get the benefit of HA/DR here, plus the capability to horizontally scale your search tier as you need to grow. Although I do like having one large beefy instance, you'll get better performance and scalability going with something like 3 x m5.8xlarge. Benefits here are numerous, and with ES I am going to assume this is for S/NOC which will have some HA requirements that a single instance will most likely not adhere to..   As always, a deployment at this scale has a lot of moving parts, and the above are just some general high level talking points without knowing the full depth of the environment. You definitely should involve your Splunk SE / Sales team and ask for sizing assistance on this. We have some internal tools that will help provide a more accurate picture of what you need. ... View more