How Splunk indexers operations works when it comes into manual detention state ? We are migrating from RHEL 6 - RHEL 8 and here can't do OS upgrade on the same machine , so we will be getting new RE...
See more...
How Splunk indexers operations works when it comes into manual detention state ? We are migrating from RHEL 6 - RHEL 8 and here can't do OS upgrade on the same machine , so we will be getting new REHL 8 machines and then we need to install Splunk there and work accordingly. Current setup : 5 indexers (RHEL 6) in indexer cluster , S.F -1 , R.F -1. Our plan is to add new 5 indexers (RHEL in indexer cluster and route all sources to send data to newly added indexers , once that is complete will make sure any of the source is not sending anything to old indexers. Enable manual detention on old indexers so that they dont replicate their data to other indexers (newly added), was going through manual detention documentation (below) and found only four operations are concerned if peer is in manual detention. stops replicating data from other peer nodes. optionally stops accepting data from the ports that consume external data, causing the peer to no longer index most types of external data. continues to index internal data and stream the data to target peer nodes. continues to participate in searches. https://docs.splunk.com/Documentation/Splunk/8.0.5/Indexer/Peerdetention But I would like to know if peer is in manual detention will it be rolling over the stored indexes data as per retention policy to frozen DB ? Or except above 4 points, all other operation will be as usual even if peer is in detention mode. Why am I concerned because in my case we have hot DB (local mount - 700 GB data on each peer) and cold DB (NAS mount - 2.3 TB each peer) and we are going to share same mount for new peers as well with creating separate subfolders from NAS (for new peer). Second thing how my S.F and R.F behaves since its 1:1 , because most of the data will be with old peers and those would be in detention mode , are S.F & R.F gonna met , is there any impact in searches. Final step would be run command splunk offline --enforce-count to decommission old peer one by one , but that all depends on above questions(if peer is in manual detention will it be rolling over the stored indexes data as per retention policy to frozen DB ?) , if peer (enabled detention mode) is not purging the data as per retention period then we would not have enough space in NAS storage (cold DB) to store the data which is coming after rolling over form hot DB from new peer. If old peers (enabled detention mode) continue to purge their data as per retention period in that way will have space cleared from one side and new data getting added to NAS (cold DB) from another side. If all this works , i dont know if R.F is set to 1 means each peer will have one copy for data, if some peer doesnt have , i think they will create new replicate copies when we do decommission. Please suggest