Multi-site using one-way replication from ICS enclaves questions.

[BrianAbbott]

We have an interesting challenge before us. We have numerous firewalled enclaves that are also prone to frequent network outages. Due to the geographically dispersed and harsh environments (truly), it is possible for network drops to persist for days. Each enclave has a mix of legacy computers and ICS data sources.

These enclaves are necessarily firewalled from our "working LAN". The enclaves do connect to a "parent" DMZ though, and that network can communicate with the LAN. We brainstormed the following strategy, but would very much value some thoughts from the Splunk community. This is entirely new territory for us.

Our thoughts...
- Each enclave will have an indexer which will also serve as a search head. They will be sufficiently resourced.
- There will be 15 to 20 enclaves (site1, site2 ... site15, etc), each with its own unique group of data sources. Time before frozen will be set to 30 to 60 days (to tolerate network outages and have most current events searchable at the location).
- There will be a "parent" DMZ network which will also host a Splunk indexer and a search head, both of which will be well resourced. This will be site0
- We want to replicate data from the enclaves (site1, site2, and so on) to the site0 indexer. We do not think that this is going to exceed 50GB per day, in fact probably closer to 20GB daily.
- We use a LM provided to us by another "authority" but we do not have direct access to administrate it. We currently have a mature Splunk environment on our working LAN that connects to it. Unfortunately, we are not sure how to configure the proposed sites to the LM.

Technically, we don't yet know how to implement this strategy or if it is even feasible.

Questions we have for the Splunk community include:

1. Is this a feasible approach? Are there better strategies? We considered using UFs with ample storage but that does not solve the problems of having capacity for a multi-day outage (potentially a couple of weeks) nor making the data searchable within the enclave.
   
2. Would it be wiser (or necessary) to use a HF instead of an indexer at the enclave level to store and forward during outages, or is the indexer/site# doable?
3. Are we able to do this without clustering the sites?
   
4. How would be best handle connecting to the LM? Site0 will be able to connect to the LM. However, none of the enclaved sites will be able to reach out to the LM.
5. What would the one-way configuration even look like from sites1-x going to site0?

I realize that this is a lengthy post with lots of info and some questions but if you would have some thoughts or if you have similar experience that could be highly valuable to us and our efforts.

Thank you.

[skalliger]

Hi there,

hm, this really is the sort of questions you would want to ask PS. Because before you can answer any of this and someone actually starts thinking about your scenario, there will be quite some questions coming up. If you're planning to do such a deployment, getting PS to give you an optimal approach is what I would aim for.

I see a few points here that need to be discussed, like the IDX or HF question, but this isn't done by saying "take option 1" without further information.

Only one thing to consider. If you can't connect all your instances to your LM, get in contact with Splunk support. They can split your license, which I would do and have done several times for companies that have multiple Splunk deployments that can't or aren't allowed to communicate with each other.

Skalli