Splunk migration: Tips for moving data, saved searches, and reports?

[dpadams]

I've been experimenting with Splunk for a few weeks and now have nine machines working as light forwarders sending several logs each over TCP. So far, so good - but Splunk is quite slow on the target server. Before I try and move the installation to a new machine, I'd like to double-check that I understand the steps. Briefly:

1. Shut down Splunk on the old server, new server, and forwarders.
2. Copy all or part of /Splunk/var/lib/splunk/ to the new machine.
3. Reconfigure my customized outputs.conf to point to the new server.
4. Distribute the updated outputs.conf to the forwarders.
5. Start the new server and see that it looks okay.
6. Restart the forwarders and see that they look okay too, either by checking the splunkd.log locally or by checking that they all seem to be posting events to Splunk.

If I've got anything wrong there or have skipped as step, I'd be grateful for advice. Also, I'm not sure of the following:

• What exactly do I need to copy from /Splunk/var/lib/splunk/? Everything?

• Where are custom searches and reports stored? I don't have many but figure I should sort this part out. I've saved these through the Splunk Web GUI and have not hand-edited, moved (or even found) any savesearches.conf files or the like.

• We're on Win32 right now and are contemplating moving either to a fast Win machine or a Linux machine. Any serious gotchas to be aware of there?

Thanks again for any help.

[dpadams · Accepted solution]

Just to close this question out, I thought I'd report back. Our Splunk server is on a virtualized copy of Windows so we imaged the machine, upgraded it and restarted. That took less than half an hour and at the end of it, only the IP address had changed. So, in this particular case, the checklist was:

1. Shut down Splunk on the old server, new server, and forwarders.
2. Reconfigure inputs.conf on the new server to match the new machine's address.
3. Start Splunk on the new server, log in and check that the data and other elements are there.
4. Reconfigure outputs.conf to point to the new server's address.
5. Distribute the customized outputs.conf to the forwarders.
6. Restart the forwarders.
7. Check that the forwarders are working, either by checking splunkd.log locally or (what I did), letting it run for a bit and then checking the last fifteen minutes of events. All of the forwarder host addresses should show up.

It was pretty quick and painless, but this is about a simple an upgrade as you could hope for. Specifically, no files were actually moved and all of the forwarders are reading from physical logs, so no ephemeral events are lost during the downtime.

Getting Splunk onto a better piece of (virtual) hardware has really improved the search speed on our roughly 8,000,000 events.