SPLUNK Architecture Deployment Minimal (Recommendations)

[willadams]

We currently use a single SPLUNK Enterprise server that runs on a single virtual machine on ESXi. This instance is both our search and index device. It has been running quite solidly for a while now, but we are looking at a way to effectively provide DR/HA as this will likely become our SIEM in the long term. The single instance has SSD disk and has 8 vCPU dedicated to the machine (CPU's are Intel with a clock speed of 3.4Ghz).

Every time I look at redesigning this server for DR, I end up with a design that will cost a small fortune in just the hardware alone, especially taking into consideration this may become the primary SIEM over time. I need the server to be highly available as during a incident this becomes critical for us.

I would initially need 2 indexers (1 at each site) and potentially 1 or 2 search heads. However SPLUNK doco says that I must have a minimum search head cluster of 3. If I read the SPLUNK doco right and I buy say a 2 x 12 core ESXi hosts with dedicated vCPU, this alone means that I would need to purchase 4 or 5 hosts to manage this load.

This is not a financially viable option. Another thought might be to just have a single search head (not clusterd) with 2 indexers (an indexer and search head on 1 ESXi host) and then have another indexer on the DR host. This would be 2 hosts rather than 5. I could in theory then just use ESXi replication for the search head (and avoid the SRM costs and infrastructure). Would this be a viable alternative?

The amount of data we currently ingest is around 30GB, but this will ramp up quite quickly. Over time I also need to make consideration for expansion/growth. We are looking at cloud, but at the moment the focus is just on an on-premise model.

Thoughts? Appreciate any responses.

[Anam]

HI @willadams

Looks like you have a few possible solutions to your question. If one of them provided a working solution, please don't forget to click "Accept" below the best answer to resolve this post. If you still need help, please leave a comment. Don’t forget to upvote anything that was helpful too. Thanks!

[woodcock]

The cheapest HA/DR option will be to convert your storage to SS/S3 (HA cloud storage), create a stand-by, non-clustered Search Head with periodic manual sync of apps and users directories. In your case, a Search Head Cluster is overkill and definitely not worth the hassle.

[willadams]

Yes I think the single search head replicated to a standby might be the way to go. Will investigate index cluster with a rf of 2 and vm replication of an unclustered search head.

[willadams]

Can I do the following without much of an issue.

Stand up 2 indexers and a single search head. Obviously this is just splunk enterprise installed on each. So I would have

• VM with an indexer oni it
• VM with an indexer on it (this would be on seperate ESXi host)
• VM acting as a search head (this would then be replicated to DR).

Storage I was going to have on local disk (ssd) on each host.

This way I could have an index cluster with a rf of 2 so my data is in 2 spots and I just configure the search head to search across both.

[woodcock]

This should work OK but you also need a Cluster Master node.

[niketn]

@willadams have you referred to Splunk Validated Architectures. You can check out .Conf Session The Hitchhiker's Guide to Splunk Validated Architectures, for understanding what this document is all about.

[codebuilder]

The documentation is correct that you need 3 nodes for a search head cluster. That is the only way to avoid split-brain situations.
However, clustering your search heads is not required. There is no reason you can't have one to many individual search heads attached to your indexers. Though they may be doing duplicate work as your search artifacts will not be replicated.

Based on the description of your requirements I would suggest that your environment is a prime candidate for running Splunk in containers. That topology will require less hardware while allowing you to scale, minimize resources, and also realize high availability.

[willadams]

I might look into the additional unclustered search heads option and see where that leads. I did look at docker to run Splunk in containers but wasnt sure on the ha/dr options I have there (dont know the intricacies of docker)

[codebuilder]

There are many solutions for implementing container orchestration with Docker. It just depends on what you want to accomplish and what you feel comfortable with.

The most popular options would be Kuberenetes (my favorite), DC/OS, and Docker Swarm.

[willadams]

But isn't Docket only supported for an S1 type SVA? There doesn't look to have been any further updates with potentially doing something like a D1 SVA deployment. We have a layer 2 network between our primary and secondary site, so conceivably I could look at this as a single site deployment and have 2 indexes clustered potentially with a single search head. The search head could just be replicated. I will try and experiment with docker and see where that goes (although this will be a new implementation of something unknown but be good from a learning exercise).