What is the best approach for upgrading SPLUNK?
running 6.5.1 on link rehl 6
download 6.63, copy to each device and then untar or is there an upgrade from the UI?
Do I need to worry about backing up custom stanzas?
at first I suggest to upgrade to the latest version of Splunk!
in addition if you haven't a cluster there isn't a real upgrade order: the only best practice is that Indexers version should be greater than Universal Forwarders one.
I usually upgrade before Search Heads, Indexers, then Deployment Server and at least Universal Forwarders.
Upgrade must be done only by CLI not by UI.
To backup the full installation is always a best practice suggested by Splunk before upgrading.
is backing up the full installation making a copy of $splunkhome/etc and all sub folders or does this include everything under $splunkHome?
making a copy of all $SPLUNK_HOME you can restore your installation in less time, instead backing up only $SPLUNK/HOME/etc to restore your installation you have to reinstall the old version and then override etc folder.
It depends by your feeeling of a good conclusion of your job: at least backup $SPLUNK/HOME/etc .
FYI, we've posted an upgrade roadmap with links to the latest documentation to help with upgrade planning. Check it out and let us know if you find it helpful. What's the order of operations for upgrading Splunk Enterprise?