Request for Guidance on Running PROD Index Servers in another site without Cold Storage

[Nraj87]

I would like to run a copy of  PROD Indexer servers’ VMs in another site (DR setup) without mapping Cold Storage, to validate environment readiness.

Could you please advise if any of the below approaches are suitable, or if there are other recommended options?

Option A -Take VM snapshot or backup copy of PROD indexers.

Deploy in test site without mounting cold storage disks.

Before starting Splunk, edit /opt/splunk/etc/system/local/indexes.conf (or deployment-specific conf path):

• Hash out/remove coldPath entries for each index.

• Splunk will start and treat the indexes as having only hot/warm storage.
  
  ========================================================================================================================================================

Option B – OPTIMISTIC_ABOUT_FILE_LOCKING = 1

• What it does: This setting tells Splunk to ignore file lock checks during boot (commonly used on NFS mounts or shared storage).
• Pros:
• Indexers will start even if file locks are not available.
• Let’s bring up DR indexers without cold storage attached.

Impact: Unknown

================================================================================================================================================================

Option C – Temporary Cold Storage + Increase maxWarmDBCount

• What it does:
• Attach temporary 200GB cold storage.
• Increase maxWarmDBCount from 300 → 1000, which allows more warm buckets to be retained before rolling to cold.
• Pros:
• Keeps logs warm longer (delays need for cold storage).
• Reduces immediate cold storage dependency during DR migration.

[PickleRick]

It's not obvious for me what is your goal. "Validate environment readiness" to do what? If this is a test of DR procedure, without cold you don't have full DR.

Normally you can run without cold (but you still have to have it defined) when running with SmartStore - then your hot/warm storage is hot/cache and the "cold" data is stored in Smartstore.

But you still need coldPath defined and accessible.

Of course you might point your server to an empty directory in which case you'll simply lose your previously-indexed-elsewhere data. So this is not a full DR if you do that - you've already lost much of your data.

So what is your actual use case?

[Nraj87]

We are building new Cold Storage (ColdDR) at the DR site, with data replicating from the PROD SITE(ColdProd).

Before performing the actual DR, I’d like to test whether my Index Server can boot at the DR site using a temporary Cold Path (TempColdDR).

I have two questions:

If the server is brought back to the PROD site with the actual Cold Storage (ColdProd), will any old data be lost?

Is there a way to pause data movement from Warm to Cold during the reboot?

[PickleRick]

You're still describing something which probably is obvious to you but it's not clear to me (and possibly others). We don't know your environment so your terminology might not be easily understandable.

As I understand it - you have several indexers running in VMs.. You're replicating something to another site. We have no idea what it is and how it's replicated - block storage? file storage? How is this storage related to the original VMs? You're also asking about a "copy" of your indexer VMs. How are those copies made? How do they relate to the storage replication of which you are speaking of?

BTW, when the server is rebooting, splunkd process is not running so no operations on splunk data is being performed.

[livehybrid]

Hi @Nraj87 

Im confused by the two options B here so assume the second is option C and not an extension of the first B!

Its also worth noting that this isnt a typical way of operating Splunk - if you want a DR backup then you should look at a multisite cluster. Please see  https://help.splunk.com/en/splunk-enterprise/administer/manage-indexers-and-indexer-clusters/9.4/ove...for more info.

Specifically answering your question though...

Based on the docs, Splunkd does not start if an index lacks a valid 'coldPath'. therefore Option A is not possible.

Option B sounds like its using coldPath that doesnt exist - The docs suggest a 'valid' coldPath which might mean that checks are made to the paths existence, regardless of the file lock checking, therefore I do not think this would work either. 

That leaves the 3rd option (C) as the remaining viable solution. 

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing