How to upgrade Search head pooling in upgrading Splunk from 6.0.1 to 7.2.3

[ramprakash]

Hi,

I need urgent assistance on upgrading Search head pooling. Mine is distributed environment(6.0.1) with below details

Two indexers(Clustered)
Two search heads(SHP)
One Cluster master

As per the Splunk docuemntation I need to upgrade in below sequence
Licence Master ->Search head ->Cluster master ->Indexer

For Search head pooling i have below doubt as mentioned in Splunk documents

Test apps prior to the upgrade
Before you upgrade a distributed environment, confirm that Splunk apps work on the version of Splunk Enterprise that you want to upgrade to. You must test apps if you want to upgrade a distributed environment with a search head pool, because search head pools use shared storage space for apps and configurations.
When you upgrade, the migration utility warns of apps that need to be copied to shared storage for pooled search heads when you upgrade them. It does not copy them for you. *You must manually copy updated apps, including apps that ship with Splunk Enterprise (such as the Search app) - to shared storage during the upgrade process*. Failure to do so can cause problems with the user interface after you complete the upgrade.
On a reference machine, install the full version of Splunk Enterprise that you currently run.
Install the apps on this instance.
Access the apps to confirm that they work as you expect.
Upgrade the instance.
Access the apps again to confirm that they still work.
If the apps work as you expect, move them to the appropriate location during the upgrade of your distributed environment:
If you use non-pooled search heads, move the apps to $SPLUNK_HOME/etc/apps on each search head during the search head upgrade process.
If you use pooled search heads, move the apps to the shared storage location where the pooled search heads expect to find the apps.

My Question is

1) I have already apps placed on NAS. How can i copy and paste from Search head again ? Is this makes sense ?

PS:- I know Search head pooling is depreciated feature. We will upgrade to Search head clustering later as a different project.

[woodcock]

I assume that you mean that you have new versions of the apps staged on your NAS, ready to move to $SPLUNK_HOME/etc/apps. Once you bring all of your search heads down and have upgraded 1 Search Head, bring it up test it with the old apps, then upgrade the apps on the NAS and test the upgraded Search Head with the upgraded Apps, then upgrade the rest of the Search Heads.

[ramprakash]

Hi. I wanted to ask that how can i copy the upgraded apps from splunkhome/etc/apps to NAS ..suppose Search app is common in both the places but Splunk takes configuration from NAS(Search head pooling). Now the upgrade will not happen in NAS as per Splunk document..How can i move the upgraded app to NAS then ..they will overwrite knowledge objects also

[woodcock]

Is this a trick question? It is a NAS. Mount it multiple places and use cp to copy files.

[sloshburch]

Copy it the normal way you'd do it on the operating system.

If you've been set up correctly, the search app on the pool will have a local folder with knowledge objects while the one in $SPLUNK_HOME/etc/apps/search will not have a local folder (or at least won't have any knowledge objects...it may have a nearly empty app.conf file).

Therefore, nothing should be lost.

[jnudell_2]

What is the purpose of your search head pooling? Do you have more than 10 users at any given time running Splunk searches? What is the hardware specification of the search heads (CPU, memory, disk space)?

[ramprakash]

Yes we have more than 10 users running at a time.
Below are the specifications

CPU- 12 Core
Ram - 16 GB
Disk space - 500 GB

Do you have steps to upgrade SHP ?

[gcusello · Accepted solution]

Hi ramprakash,
as you said Search Head Pooling is a deprecated feature, but I didn't find any information about the version of removal.
Anyway, I think that you don't need to copy apps again because you already have them on NAS.
I had a very bad experience with Search Head Pooling upgrade, so if you don't need to upgrade now, maybe it could be better to wait and upgrade when you'll pass to Search Head Cluster.

Bye.
Giuseppe

[ramprakash]

@gcusello Thanks for sharing your experience. Unfortunately I can`t wait to upgrade to SHC. Did you follow the below procedure as mentioned in Splunk docs to upgrade SHP. I have confusion on point 8. Why is it asking to copy the apps again ?

Upgrade the search head pool
Caution: Remove each search head from the search head pool before you upgrade it, and add it back to the pool after you upgrade. While you don't need to confirm operation and functionality of each search head, only one search head at a time can be up during the upgrade phase.

Bring down all of the search heads in your environment. At this point, searching capability becomes unavailable, and remains unavailable until you restart all of the search heads after upgrading.

Place the confirmed working apps in the search head pool shared storage area.

Remove Search Head #1 from the search head pool.

Upgrade Search Head #1.

Restart Search Head #1.

Test the search head for operation and functionality. In this case, "operation and functionality" means that the instance starts and that you can log into it. It does not mean that you can use apps or objects hosted on shared storage. It also does not mean distributed searches will run correctly.

If the upgraded Search Head #1 functions as desired, bring it down.
8. Copy the apps and user preferences from the search head to the shared storage.

Add the search head back to the search head pool.

Restart the search head.

Upgrade the remaining search heads in the pool with this procedure, one by one.

[ramprakash]

@gcusello .. Could you please assist with point 8 above

[sloshburch]

Step 8 ensures that the apps that come with Splunk (like search & reporting and some behind the scenes apps) get's updated.
After you upgrade, the latest will be in $SPLUNK_HOME/etc/apps (by default) so copy and paste them into the pooled location which essentially upgrades the apps in the pooled location.

Remember to copy and paste them over the apps of the same names already there in order to preserve any knowledge objects in the local directories.

[ramprakash]

@SloshBurch .. Hey Thanks for explaining so it means i can copy them over the apps of the same name like SHPOOLING/apps/Search/Search ??

[sloshburch]

Yea, copy over. It's like upgrading in place. For example, copy the $SPLUNK_HOME/etc/apps/search app to the same in NAS. Make sure you copy correctly or you'll end up with NAS/search/search, which means copied the search app INTO, not on top of, the NAS location.

You won't see the changes take affect until after restart of the search heads reading the NAS location.

[ramprakash]

Hey Thanks! I got this. Could you please confirm if i understood correct.

Present Apps in $SPLUNK_HOME/etc/apps

drwxr-s---+ 4 splunk splunk 4096 Jun 27 2017 user-prefs
drwxr-s---+ 3 splunk splunk 4096 Jun 27 2017 legacy
drwxr-s---+ 6 splunk splunk 4096 Jun 27 2017 framework
drwxr-s---+ 9 splunk splunk 4096 Jun 27 2017 search
drwxr-s---+ 5 splunk splunk 4096 Jun 27 2017 learned
drwxr-s---+ 6 splunk splunk 4096 Jun 27 2017 gettingstarted
drwxr-s---+ 6 splunk splunk 4096 Jun 27 2017 sample_app
drwxr-s---+ 6 splunk splunk 4096 Jun 27 2017 launcher
drwxr-s---+ 4 splunk splunk 4096 Jun 27 2017 SplunkLightForwarder
drwxr-s---+ 4 splunk splunk 4096 Jun 27 2017 SplunkForwarder
drwxr-s---+ 4 splunk splunk 4096 Jun 27 2017 splunk_datapreview

Present apps in NAS (SHP)

drwxr-s---. 5 splunk splunk 4096 Mar 7 2014 user-prefs
drwxr-s---. 6 splunk splunk 4096 Mar 7 2014 launcher
-rw-r-----. 1 splunk splunk 0 Mar 7 2014 sentinel.txt
drwxr-s---. 4 splunk splunk 4096 Mar 7 2014 learned
drwxr-s---. 10 splunk splunk 4096 Jul 25 2014 search
drwxr-s---. 4 splunk splunk 4096 Jul 25 2014 admin
drwxr-s---. 6 splunk splunk 4096 Oct 9 2014 sideview_utils
drwxr-s---. 9 splunk splunk 4096 Dec 8 2014 sos
drwxr-s---. 10 splunk splunk 4096 Jan 9 2015 integration_platform
drwx--s---. 5 splunk splunk 4096 Dec 20 2017 base64
drwxr-x---. 2 splunk splunk 4096 Jun 13 2018 sara
drwxr-xr-x. 9 splunk splunk 4096 Jul 1 10:44 rbcone_registry

So i need to copy updated apps of $SPLUNK_HOME/etc/apps to NAS under the same app name .

So if in NAS we have search APP . Do i need to copy search directory from etc/apps to NAS/apps/search. Is it correct ?