Can you help me migrate from Splunk 6.6.0 to Splunk 7.2.1?

[cadm777]

I am a new Splunk user. I have finished all the Splunk Admin training. (Splunk Fundamentals 1 & 2, Splunk Administration, and Splunk Data Administration) I am still processing all the information.

However, I've been tasked with developing a migration plan to move our current environment from Splunk 6.6.0 to Splunk 7.2.1. A fairly large task for a new Splunk user. I'm excited about it, and not afraid to take on the task. So just need some help in getting pointed in the right direction. For security reasons, I can't give out to any details, but I'll include what I can.

Splunk 6.6.0 environment:

• 1-Windows 2008 Deployment Server: --serverclass.conf file location and deployment clients
• 1-Windows 2008 Index Controller Server: --Serves as the main index controller and license manager, where all non neteng users log in to search ingested data.
• 1-Linux OS Traffic Indexer --Neteng syslog traffic indexer
• 2 Linux OS Lightweight Forwarders --Running syslog-ng, the original syslogs from network devices get pushed to the indexers from here I think.
• 1-Windows Server 2008 Search Head --Searches the neteng data --All neteng users login here to search neteng data

What I need to gather from what I know:

1. A complete list of the servers from the "white-list" and "blacklist" (I think I can get this from the\Program Files\Splunk\etc\system\local\serverclass.conf file. Just not sure if it is the complete list.)
2. File types and their paths that are being monitored on these servers (From the deployment server "Settings" > "Data Inputs" > "Forwarded Inputs"??? Is there an export function for this?)
3. User data including alerts, reports, and dashboards. Need to know the owners of each.(Need to know how to export this, and then import to the new environment)
4. Is the user data private or global? (Not sure where to get this from)
5. Improve the efficiency of the saved searches and reports (This is something we can work on here. Is there a search analyzer in Splunk?)
6. Who has access to the apps? (Group description + matching AD group) (Is there a way to compare the AD list against the Splunk users? I need to create a spreadsheet of the users, their Splunk roles, and the AD group they belong to. Is there an export function of user roles?)
7. Index data migration (Index information - "Settings" > "Indexes". Is there a migration plan for moving indexes from one environment to the other?)

Once all data is gathered, our goal is to ensure that users aren't having to straddle both environments to access their content. So, this means migrating a group with their index, and migrating the searches, reports, and alerts for that index.

If anyone has a basic migration project plan, it would be helpful for the newbie.

Thanks for your help.

[cadm777]

01/10/2019 UPDATE:
I was told this week by management that we will not migrate any of the indexed data. We are only going to migrate the AD Groups, Roles, users, capabilities, and user's data. We will also move the associated indexes, if they do not exist in the new system. So for now, I just need to match up the AD Group with the role and capabilities for the users, and the indexes associated with them. Also need information on how to migrate the data to the new system.

Apparently we only keep a few weeks of indexed data. So we are working on moving the users with the index and servers and logs at the same time, and then point the users to the new environment. No need for bucket moves.

[richgalloway · Accepted solution]

What you describe is more of a merger than a migration. While it's possible, it's more complicated. The main thing to look out for is bucket collisions. See https://docs.splunk.com/Documentation/Splunk/7.2.3/Installation/MigrateaSplunkinstance

To answer your other questions:
1. The serverclass.conf file will contain all of your forwarders, but won't contain all of them. Unfortunately, there's no one place that has all instances. See https://docs.splunk.com/Documentation/Splunk/7.2.3/InheritedDeployment/Introduction for tips on finding your Splunk server types.
2. The best way to get this information is to look at the input.conf files on each forwarder. Shouldn't be a problem since you only have two.
3. The GUI can show you this information. Export, however, is probably limited to printing to a PDF. Another option is to use REST. Search for | rest /serviceNS/-/-/saved/searches to get a list of all saved searches. Filter by adding search commands like `| search NOT eai:acl.owner=admin'. This can be exported.
4. If it's in $SPLUNK_HOME/etc/users it's private.
5. A search analyzer would be a great feature. There are probably apps which can help with this, but I can't think of any ATM. If you want to write your own, REST is the key. You can use the search from #3 to get the bodies of all searches can search them for key attributes of inefficient searches, like "index=*", "join", "transaction", etc.
6. I'm not sure about this one, but REST probably is a big part of the answer.
7. See the Migrate A Splunk Instance link above.

[cadm777]

Rich,
Thanks for the response. This really helps, and gets me pointed in the right direction. I figured I would have to write some rest searches for most of it. This is going to be a complex task for sure. Lots of data to pull together. I did find information on the "Data Governance" app.

https://splunkbase.splunk.com/app/1866/

I'm not sure if that will help, but I'll get it installed, and give it a try.

Thanks again.

[richgalloway]

When you say "migrate" do you mean you will be installing the new version of Splunk on the existing servers or on new servers? If the latter, why? What will the new platform(s) be? If you will be using new servers, consider an all-Linux environment. You'll thank me later.

[cadm777]

The new Splunk 7.2.1 is installed and up and running. It is an all Linux environment. Think of it as a completely separate environment. It is processing other forwarders now. We want to completely decommission the 6.0 environment all together.

Current New Environment:(Not really new, just a new enterprise environment and all Redhat Linux)
2-lightweight forwarders (For syslogs only)
1-Heavy forwarder
1-Dev box for testing heavy forwarders
1-Splunk ES server
2-Deployment servers: One used to push updates to the non ES search heads. The other one is where all the apps live. The second one pushes updates to the cluster master, deployer, light forwarders, heavy forwarders, and universal forwarders. It is also has the common applications directory.
3-Search head servers as a cluster
1-Cluster master that manages the index servers
15-index servers as a cluster