Getting Data In

migrate data from single splunk indexer, split to two new indexers (sanity check)


I have a plan to migrate data from a single splunk indexer to two separate indexers, reconfiguring the production system from Solaris to RedHat in the process. I've done some testing and it looks like this will work, but need a sanity check. If there are flaws in what I'm proposing let me know... Thanks.

Current environment:

Splunk indexer / web

  • 5TB SAN partition
  • 24 GB RAM
  • Splunk 4.2.5
  • single index (main/defaultdb)
  • Solaris 10 / Intel x64

Phase 1

  1. Bring up second Splunk Indexer
    1. 24GB RAM
    2. Splunk 4.3.1
    3. 3TB SAN partition mounted at /opt/splunk
    4. 3TB SAN partition mounted at /opt/splunktmp
    5. RedHat 6 Enterprise, x64
    6. create "migrate" index in default location, "migratetmp" index in /opt/splunktmp/var/lib/splunk/
  2. copy db_* directories in existing defaultdb ending in an odd number to migrate/colddb, even numbers to migratetmp/colddb:
    rsync -av --progress --stats --rsync-path /opt/sfw/bin/rsync splunk@oldsplunkserver:/opt/splunk/var/lib/splunk/defaultdb/db/db_*{1,3,5,7,9} /opt/splunk/var/lib/splunk/migrate1/colddb/
  3. point all splunk forwarders to new splunk server... no new data to old splunk server
  4. do a final roll of hot to warm on old splunk server, shut down splunk on both servers
  5. do a final rsync to pick up the final bits of log goodness in the old splunk server

Phase 2

  1. Reconfigure old splunk server as a mirror of the new server
  2. detach 3TB partition from the phase1 server mounted at /opt/splunktmp, attach to new server at /opt/splunk
  3. rename migratetmp directory to migrate
  4. configure splunk forwarders to load balance between the two servers

End result should be:

  1. New data coming in in the main/default (or otherwise appropriate index)
  2. Old data available in the migrate index
  3. data will age in the migrate index as defined as time moves on
  4. searches are faster, less storage needed than otherwise (would have to keep an additional 5TB partition for the duration of the data's life), etc.
1 Solution




Thanks for the confirmation... I had already read the documentation and ran some tests on my own, so I was pretty confident already. My constraint in my situation is the limitation of the two servers. The current production system is going to be refreshed and changed from solaris to redhat, so I don't have the luxury of simply having two servers to move to right off.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!