Is there a Moving Splunk enterprise to new system (windows) step-by-step guide?

[kjmcc123]

Good Morning All,

I'm having a hard time moving the entire C:\Program Files\Splunk folder to a new system. I've seen the "guide" online but it just says move the Splunk Home folder. IS this the same thing as the entire Splunk folder?

My main goal is to get the old logs showing up on the new system. The  C:\Program Files\Splunk folder is about 100 gigabytes. I receive an error when trying to zip the folder or transfer to a NAT. (server.pem not allowed). 

Anyone have a step-by-step on what I need to do for this to work? Do I just need to transfer a particular folder? I only use the default/main index for data. I'm on version 6.xx for enterprise. 

Please help! 

- Kevin

[Stefanie]

Yes the C:\Program Files\Splunk folder is essentially  your "Home" folder. 

We migrated the Search Heads, Cluster Master, Deployment Server, and Indexers to new servers by coping over the Splunk folder. (Albeit, ours was Linux so '/opt/splunk')

Personally we decided to upgrade our version of Splunk to the latest version so that we would be able to receive assistance from Splunk if need be.

How many servers are you migrating and what are their roles (Search Head, Indexer, Cluster Master, etc)?

[kjmcc123]

just one server and an indexer. Thanks for your help

[Stefanie]

These are the instructions I followed from Splunk when I did my upgrade:

Spot check Search Head functionality 

Migrate Core Splunk Search Head

            Stop Core Splunk SH

            Straight copy Splunk folder to new Core Splunk SH (This is the C:\Program Files\Splunk folder)

            Ensure licensing configuration was migrated

            Modify server.conf -

                        under [general] stanza

                        Set servername to new server.

                        clear text on pass4symmkey {default is changeme}

                        under [sslConfig] stanza

                        clear text sslPassword {default is password}

                        under [clustering]

                        clear text on pass4symmkey

            Insert latest Web Console certificates (Prepare certificates)

            Start new Core Splunk SH

            Validate SH functionality

Some of it  might not apply for your environment such as the clustering stanza. 

We had a indexer cluster so migrating an Indexer was easy. But since you have only one indexer the steps here https://docs.splunk.com/Documentation/Splunk/8.2.5/Installation/MigrateaSplunkinstance seem easy enough 

Roll any hot buckets on the source host from hot to warm.
Review indexes.conf on the old host to get a list of the indexes on that host.
On the target host, create indexes that are identical to the ones on the source system.
Copy the index buckets from the source host to the target host.
Restart Splunk Enterprise.

Hope this helps and gives you a bit more guidance.

[kjmcc123]

Okay we'll try this. I changed the index.conf file (all it had was default, "Server name" below it) not sure if this is the correct index.conf file (there seem to be many in different locations?) I think mine was at c:\program files\Splunk\var\local , I changed the system name to the system I plan on installing Splunk on. 

We having a very basic barebones install....Splunk enterprise default everything and all we ingest is application, security, system, forwarder event logs (that's the default configuration that Splunk applies) and we configure it to use receiving indexer (no deployment servers) and port 9997 (default) on enterprise receiving indexer port #. 

Then we have our dashboards to filter and display the data we want for review. 

Does it matter that the location of the Splunk folder will change from c:\program files\ to a data drive -->  d\Splunk (a storage drive ) 

When I do the Splunk install of enterprise I will change the location to the D drive (where we copied the initial c:\program Files\Splunk folder from original machine) and hopefully it will work. If you have an idea of the file location (default location) of where that index file is that would be helpful.

The main reason we are doing this is that our Splunk server is currently on a workstation with a 256gb HD and our Splunk folder is 100 gigabytes, so we want to move it to a file server data drive and have it operate off that. 

Thanks again! 

[Stefanie]

That should be fairly simple. 

Depending on your original installation you may have some conf files placed in different directories. 

Sound's like it might be less of a hassle if you freshly install the same version of Splunk you are running currently onto the D drive, let it start up to create the default conf files, and then stop splunk and copy over your configurations/apps.

I'd start by copying everything from C:\Program Files\Splunk\etc\apps\* to D:\Splunk\etc\apps 

C:\Program Files\Splunk\etc\licenses\* to D:\Splunk\etc\licenses

and copy over C:\Program Files\Splunk\etc\users\* to D:\Splunk\etc\users

After those three directories are copied over, you will need to copy the conf files in C:\Program Files\Splunk\etc\system\local  ... You can not do a straight copy over as some of these files are specific to the directory they're installed in. 

My suggestion is to open each one and bring over the configurations that do not mention the old Server name or a password. Ensure that the configurations to connect to your Indexer are brought over.

[kjmcc123]

is there a lot of .conf files?, I wish there was a list of the "default .conf" files that need to be copied over including the paths on windows would make this so much easier.  This part is still confusing me, so it seems that I cannot copy the entire Splunk directory, and I need to take out specific file paths?  after install of Splunk to D drive. 

I don't use any apps, I do have licenses and users.  What about dashboards?

So my process now based on your feedback will be 

1. Upgrade the original splunk server using the guide Splunk-8.2.5-Installation.pdf (It requires me to upgrade to 7.2 --> 8.0 --> 8.2) hoping this doesn't break my splunk server!

2. Install the 8.2 version to my D:\drive, stop splunk (on both ends?) and then copy over the specific directories from old splunk into new splunk on the d drive. 

What about copying over the data part? I think the data is stored at c:\program files\splunk\var\lib\default.db\db (something along these lines, its the largest folder in the Splunk directory) 

Will I be able to just copy the entire var folder over? 

Thanks for your help!

[Stefanie · Accepted solution]

There shouldn't be that many conf files. Maybe 10 or so. It really depends on your environment. 

If you can tell me the names of the conf files you have in that directory I can explain them to you.

There are some that might be specific to your current installation which is why it's best if you open each one and look at it before bringing it over, if you're new to this. 

Your dashboards are contained in apps or in user directories 🙂

The upgrade path from 7.2 -> 8.0 -> 8.2 is straightforward. After upgrading to the next step, ensure that Splunk starts and check for full system functionality. (IE, is data still coming in?,  Can you still search?, Your Dashboards still work?) 

Is your Indexer moving to the D drive as well?  Then yes you will most likely need to bring the Indexer files over too. After rolling all your buckets from hot to warm you can copy everything in the $SPLUNK_HOME/var/lib/splunk folder to your new installation. 

[kjmcc123]

You have been very helpful! 

The indexer is moving to the D drive. 

I'm not sure what you mean by rolling from hot to warm (is this done automatically or will I need to do something first?)

IF it is......for the $SPLUNK_HOME/var/lib/splunk (just the highlighted portion will be copied?) or will I be able to go one directory higher (copy the entire lib folder over and replace)

[Stefanie]

You will need to run the command 

splunk _internal call /data/indexes/<index_name>/roll-hot-buckets –auth <admin_username>:<admin_password>

 This must be done for every index you have.  This will roll your indexes to warm. 

Read the section about Working with the CLI on Windows here https://docs.splunk.com/Documentation/Splunk/8.2.5/Admin/AbouttheCLI

These are the steps to migrate the indexes again 

How to move index buckets from one host to another


If you want to retire a Splunk Enterprise instance and immediately move the data to another instance, you can move individual buckets of an index between hosts, as long as:

When you copy individual bucket files, you must make sure that no bucket IDs conflict on the new system. Otherwise, Splunk Enterprise does not start. You might need to rename individual bucket directories after you move them from the source system to the target system.

1. Roll any hot buckets on the source host from hot to warm.
2. Review indexes.conf on the old host to get a list of the indexes on that host.
3. On the target host, create indexes that are identical to the ones on the source system.
4. Copy the index buckets from the source host to the target host.
Restart Splunk Enterprise.

 Your indexes.conf (mentioned in step 2) is an example of one of the conf files you need to migrate over. This might be in your C:\Program Files\Splunk\etc\system\local directory or it could be elsewhere. Taking an inventory of the conf files and their locations will help you in the long run.

And for your question: 

---

IF it is......for the $SPLUNK_HOME/var/lib/splunk (just the highlighted portion will be copied?) or will I be able to go one directory higher (copy the entire lib folder over and replace)

---

Sure you can copy the entire lib folder if you wish. On my installation the only folder under 'lib' is 'splunk' 🙂

[kjmcc123]

so in PowerShell for instance I would CD C:\Program Files\Splunk\bin>

then run 

splunk _internal call /data/indexes/default.db/roll-hot-buckets –auth admin : password

(is this correct syntax?) I believe we only have 1 index, Is there a way on the Splunk web UI to find where the index path is?

it also appears that setting variables is optional. 

[Stefanie]

If you open up the indexes.conf file (Most likely in C:\Program Files\Splunk\etc\system\local) it will have the names of the indexes in your environment. 

Even if you mainly utilize one index, there are other indexes that Splunk uses. (_audit, _internal, for example)

Run this search in Splunk

| eventcount summarize=f index=* index=_* | dedup index | fields index

Everything in the indexes column is a type of index you have.

Example if your index is named "main" then the command you would run is 

splunk _internal call /data/indexes/main/roll-hot-buckets –auth admin : password

[kjmcc123]

I will try and grab the .conf files (would a search of .conf in the search bar of the file directory work>?) I will perform this for the entire splunk directory. 

[kjmcc123]

Also wanted to add should I upgrade the current server to the latest version first ? Then do the copy to the new system. Or do the copy first then upgrade it?