Splunk Enterprise Upgrade 7.x to 8.x

[pankajupadhyay]

Hi,

We have 2 HF and 2 Indexer in our environment which are standalone instance and running on 7.x version.

I wanna go for up gradation from 7.x to 8.x but in document of Splunk they have mentioned python 3.7 and all.

Can someone please help me with Upgradation procedure and pre-requisiste ?

Do i need to install on Python 3.7 on Linux Platform ?

[vikramyadav]

1. Test your apps and make sure they are compatible with 8.0 (you need to go to Splunk base and check if they are compatible)
2. Upgrade Deployment Server if you have(disable it first, then upgrade, do not restart it yet)
3. Upgrade Search Heads
4. Upgrade Indexers (once completed you can now restart your deployment server)
5. Upgrade Forwarders

While upgrading the indexer cluster follows steps mentioned in:
https://docs.splunk.com/Documentation/Splunk/8.0.1/Indexer/Upgradeacluster

While upgrading the search cluster follows steps mentioned in:
https://docs.splunk.com/Documentation/Splunk/8.0.1/DistSearch/UpgradeaSHC

--------------------------------------------------------

If this helps your like will be appreciated 😀

[pankajupadhyay]

@vikramyadav @isoutamo @gcusello 

Hi I have tried to upgrade my test which i have created just to test.

Step 1: Upgraded the app which support 8.0

Step 2: Configuration backup Indexer and Heavy Forwarder /etc/system/local/ and /etc/apps/

Step 3. In our environment HF and Indexer are standalone environment. 

I upgraded the Indexer and When i have checked the data was not present which i have ingested on Indexer.

hot bucket rolled to warm bucket

What did i miss ?/

If i will perform same step in production then it would be an issue as i will lose all the data which exists on Indexer.

Please help me in that.

Note: Our Indexer are not in cluster. It is a standalone system.

[isoutamo]

Usually you should backup the whole etc directory.

If/when you have followed @gcusello ‘s pointed instructions there shouldn’t be any issues. Can you give the journal what you have done and where, so we could try to help you?

[isoutamo]

Unfortunately I have found that all those apps which have said to be a compatible for 8 haven’t that. So you must down load https://splunkbase.splunk.com/app/4698/ and check those (especially python parts) by yourself and fix what is needed before update.
r. Ismo

[pankajupadhyay]

@gcusello  Thanks for the brief information.

On indexer, We have different bucket then how to actually take backup ??

System backup : Can we initiate the automatic backup or schedule backup ?

What are the precaution steps for hot bucket?

In short, What kind of backup i have to take before proceeding for upgradation ?

Thanks in advance.

[isoutamo]

Hi
you should remember that you cannot backup hot buckets!

Probably easiest way (if possible) is shutdown splunk and then take volume/filesystem level backup, if/when you want to backup you data also. If it enough to backup only configurations then @gcusello's tar is the easiest way.
r. Ismo

[gcusello · Accepted solution]

Hi @pankajupadhyay,

data hasn't any problem on update, so it's sufficient the normal backup policy you have.

It's a best practice to perform a backup (e.g. using tar) of the splunk folder, so you can immediately restore your instace if there are problems.

The most important steps are:

• to check the readiness to upgrade of your apps using the "Splunk Platform Upgrade Readiness App";
• to check the compatibility of your Universal Forwarders with Splunk 8.x: https://docs.splunk.com/Documentation/VersionCompatibility/current/Matrix/Compatibilitybetweenforwar...

Ciao.

Giuseppe

[jotne]

Unsupported does not mean that it does not work.

We have one UF 6.1.4 that sends logs to 8.1.2 Server without any problem. 

[gcusello]

Hi @pankajupadhyay,

here you can find all the informations about upgrade to Splunk 8.x:

https://docs.splunk.com/Documentation/Splunk/8.0.5/Installation/AboutupgradingREADTHISFIRST

Anyway, in few words:

• you can directly upgrade from your Splunk 7.x to 8.x;
• check the compatibility of your Universal Forwarders with Splunk 8.x: https://docs.splunk.com/Documentation/VersionCompatibility/current/Matrix/Compatibilitybetweenforwar...
• you don't need to install Python 3.7 because it's already embedded in Splunk 8.x;
• You have to set Python 3.7 on Splunk because in Splunk 8.x there are both 2.x and 3.x, but 2.x is deprecated and not more available in the next versions (https://docs.splunk.com/Documentation/Splunk/8.0.5/Python3Migration/ChangesEnterprise#:~:text=versio...);
• The first step is to install the "Splunk Platform Upgrade Readiness App" to check the readiness to upgrade of your apps;
• Then You have to upgrade before Indexers and then HF;
• There aren't specific prerequisites, only the app readiness.

Ciao.

Giuseppe

[pankajupadhyay]

@gcusello 

I have upgraded the Splunk instance and thank you for your input.

Can you please help me with one more thing ?

Even i have uploaded the latest Splunk add-on or App for Splunk Enterprise 8.1 even though it was showing me python3 warning in Splunk readliness app ?

[gcusello]

Hi @pankajupadhyay,

Good for you,

about your question, did you forced Python3?

if not follow the steps at https://docs.splunk.com/Documentation/Splunk/8.1.2/Installation/Python3LowEffort ?

in few words in  $SPLUNK_HOME/etc/system/local/server.conf set python.version=python3.

Ciao.

Giuseppe

[pankajupadhyay]

@gcusello  @isoutamo @vikramyadav 

I am about to upgrade the production Splunk servers.

Step 1: Checked the Splunk add-on and App compatabilitty and download the latest one from Splunk base.

Step 2: Configuration backup and file level system backup of Index data.

Step 3: Download the latest Splunk Enterprise 8.0

Step 4: Stop the services of Indexer first and install the latest package and restart the services.

Step 5: Will upgrade the app and add-on 

Step 6: verify eveything is working fine or not .

Step 7: Same procedure for HF then

Correct me if am missing out anything here?

[isoutamo]

That seems to be ok. Here is the official order to upgrade in distributed environment.

https://community.splunk.com/t5/Installation/What-s-the-order-of-operations-for-upgrading-Splunk-Ent...

On step 1 there may be situation that you must update those apps by yourself even those were said to compatible for 8 (at least I have updated couple of them).

r. Ismo

[gcusello]

Hi @pankajupadhyay,

yes it's correct.

Remember to force the use of Python3: https://docs.splunk.com/Documentation/Splunk/8.0.5/Python3Migration/ChangesEnterprise#:~:text=versio... after the installation of Splunk 8.x.

Ciao.

Giuseppe

[pankajupadhyay]

@gcusello @isoutamo @vikramyadav 

After upgradation on Indexer.

Do we have the possitbilities that we can loose the warm bucket as i will roll back hot to warm bucket by using command.

hot,warm,cold bucket are in default location

Frozen is on NFS.

OR do i need to restore the default db for hot,warm,cold bucket ?

Regards

Pankaj

[gcusello]

Hi @pankajupadhyay,

no, you don't need any intervene on buckets.

Ciao.

Giuseppe

[pankajupadhyay]

@gcusello  Thank you so much information.

I will go through with the information.  

[gcusello]

Hi @pankajupadhyay,

Tell me if you'll need more help.

When you'll finish, please, accept the answer for the otehr people of Community.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated 😉