Is there any documentation on safely upgrading splunk machines (master, searchhead, indexers) on splunk version 8.0.2 from RHEL 7 to RHEL 8 ?
Thank you! What are some splunk commands I can use to verify that everything is working properly? I have checked the splunk service status on all the machines, the search function in the web UI, and ran the splunk show cluster-bundle-status to check the status of the indexer cluter.
MC dashboards are a good starting point to check after update. Also do those apply xxx and rolling-restarts.
You should have some standard metrics which shows the normal situation which you should use to validate current situation.
How to upgrade your OS is a better question for the Red Hat community than the Splunk community. Splunk does not offer guidance on how to upgrade the OS it runs on.
You will, of course, need to stop each Splunk instance so it's OS can be changed. The order is not significant since you are not changing Splunk versions. Just be sure to do one instance at a time and to put your indexer cluster (if you have one) into maintenance mode first.
Splunk isn't dependent on library versions of the OS? Or, build any static libraries based on the OS?
Splunk is dependent only on the kernel version.
Any issues with the version of glibc?
Splunk's system requirements documentation (https://docs.splunk.com/Documentation/Splunk/latest/Installation/Systemrequirements#System_requireme...) mention OS kernels and a few caveats, but nothing about specific libraries. From that I conclude there are no dependencies on libraries.
Kernel and Posix threads (libpthread?)
Not sure how much the pthread library version matters.
Thanks for the input! How can we put the index cluster into maintenance mode? Do we just run the command from this document on the master node? https://docs.splunk.com/Documentation/Splunk/8.1.2/Indexer/Usemaintenancemode