I need urgent assistance on upgrading Search head pooling. Mine is distributed environment(6.0.1) with below details
Two search heads(SHP)
One Cluster master
As per the Splunk docuemntation I need to upgrade in below sequence
Licence Master ->Search head ->Cluster master ->Indexer
For Search head pooling i have below doubt as mentioned in Splunk documents
Test apps prior to the upgrade
Before you upgrade a distributed environment, confirm that Splunk apps work on the version of Splunk Enterprise that you want to upgrade to. You must test apps if you want to upgrade a distributed environment with a search head pool, because search head pools use shared storage space for apps and configurations.
When you upgrade, the migration utility warns of apps that need to be copied to shared storage for pooled search heads when you upgrade them. It does not copy them for you. *You must manually copy updated apps, including apps that ship with Splunk Enterprise (such as the Search app) - to shared storage during the upgrade process*. Failure to do so can cause problems with the user interface after you complete the upgrade.
On a reference machine, install the full version of Splunk Enterprise that you currently run.
Install the apps on this instance.
Access the apps to confirm that they work as you expect.
Upgrade the instance.
Access the apps again to confirm that they still work.
If the apps work as you expect, move them to the appropriate location during the upgrade of your distributed environment:
If you use non-pooled search heads, move the apps to $SPLUNK_HOME/etc/apps on each search head during the search head upgrade process.
If you use pooled search heads, move the apps to the shared storage location where the pooled search heads expect to find the apps.
My Question is
1) I have already apps placed on NAS. How can i copy and paste from Search head again ? Is this makes sense ?
PS:- I know Search head pooling is depreciated feature. We will upgrade to Search head clustering later as a different project.
as you said Search Head Pooling is a deprecated feature, but I didn't find any information about the version of removal.
Anyway, I think that you don't need to copy apps again because you already have them on NAS.
I had a very bad experience with Search Head Pooling upgrade, so if you don't need to upgrade now, maybe it could be better to wait and upgrade when you'll pass to Search Head Cluster.
@gcusello Thanks for sharing your experience. Unfortunately I can`t wait to upgrade to SHC. Did you follow the below procedure as mentioned in Splunk docs to upgrade SHP. I have confusion on point 8. Why is it asking to copy the apps again ?
Upgrade the search head pool
Caution: Remove each search head from the search head pool before you upgrade it, and add it back to the pool after you upgrade. While you don't need to confirm operation and functionality of each search head, only one search head at a time can be up during the upgrade phase.
Bring down all of the search heads in your environment. At this point, searching capability becomes unavailable, and remains unavailable until you restart all of the search heads after upgrading.
Place the confirmed working apps in the search head pool shared storage area.
Remove Search Head #1 from the search head pool.
Upgrade Search Head #1.
Restart Search Head #1.
Test the search head for operation and functionality. In this case, "operation and functionality" means that the instance starts and that you can log into it. It does not mean that you can use apps or objects hosted on shared storage. It also does not mean distributed searches will run correctly.
If the upgraded Search Head #1 functions as desired, bring it down.
8. Copy the apps and user preferences from the search head to the shared storage.
Add the search head back to the search head pool.
Restart the search head.
Upgrade the remaining search heads in the pool with this procedure, one by one.
Step 8 ensures that the apps that come with Splunk (like search & reporting and some behind the scenes apps) get's updated.
After you upgrade, the latest will be in
$SPLUNK_HOME/etc/apps (by default) so copy and paste them into the pooled location which essentially upgrades the apps in the pooled location.
Remember to copy and paste them over the apps of the same names already there in order to preserve any knowledge objects in the
Hey Thanks! I got this. Could you please confirm if i understood correct.
Present Apps in $SPLUNK_HOME/etc/apps
drwxr-s---+ 4 splunk splunk 4096 Jun 27 2017 user-prefs
drwxr-s---+ 3 splunk splunk 4096 Jun 27 2017 legacy
drwxr-s---+ 6 splunk splunk 4096 Jun 27 2017 framework
drwxr-s---+ 9 splunk splunk 4096 Jun 27 2017 search
drwxr-s---+ 5 splunk splunk 4096 Jun 27 2017 learned
drwxr-s---+ 6 splunk splunk 4096 Jun 27 2017 gettingstarted
drwxr-s---+ 6 splunk splunk 4096 Jun 27 2017 sampleapp
drwxr-s---+ 6 splunk splunk 4096 Jun 27 2017 launcher
drwxr-s---+ 4 splunk splunk 4096 Jun 27 2017 SplunkLightForwarder
drwxr-s---+ 4 splunk splunk 4096 Jun 27 2017 SplunkForwarder
drwxr-s---+ 4 splunk splunk 4096 Jun 27 2017 splunkdatapreview
Present apps in NAS (SHP)
drwxr-s---. 5 splunk splunk 4096 Mar 7 2014 user-prefs
drwxr-s---. 6 splunk splunk 4096 Mar 7 2014 launcher
-rw-r-----. 1 splunk splunk 0 Mar 7 2014 sentinel.txt
drwxr-s---. 4 splunk splunk 4096 Mar 7 2014 learned
drwxr-s---. 10 splunk splunk 4096 Jul 25 2014 search
drwxr-s---. 4 splunk splunk 4096 Jul 25 2014 admin
drwxr-s---. 6 splunk splunk 4096 Oct 9 2014 sideviewutils
drwxr-s---. 9 splunk splunk 4096 Dec 8 2014 sos
drwxr-s---. 10 splunk splunk 4096 Jan 9 2015 integrationplatform
drwx--s---. 5 splunk splunk 4096 Dec 20 2017 base64
drwxr-x---. 2 splunk splunk 4096 Jun 13 2018 sara
drwxr-xr-x. 9 splunk splunk 4096 Jul 1 10:44 rbcone_registry
So i need to copy updated apps of $SPLUNK_HOME/etc/apps to NAS under the same app name .
So if in NAS we have search APP . Do i need to copy search directory from etc/apps to NAS/apps/search. Is it correct ?
@SloshBurch .. Hey Thanks for explaining so it means i can copy them over the apps of the same name like SHPOOLING/apps/Search/Search ??
Yea, copy over. It's like upgrading in place. For example, copy the
$SPLUNK_HOME/etc/apps/search app to the same in NAS. Make sure you copy correctly or you'll end up with NAS/search/search, which means copied the search app INTO, not on top of, the NAS location.
You won't see the changes take affect until after restart of the search heads reading the NAS location.
What is the purpose of your search head pooling? Do you have more than 10 users at any given time running Splunk searches? What is the hardware specification of the search heads (CPU, memory, disk space)?
Yes we have more than 10 users running at a time.
Below are the specifications
CPU- 12 Core
Ram - 16 GB
Disk space - 500 GB
Do you have steps to upgrade SHP ?