Thanks for the replies. Please note that there was some planning in doing this, that the new system had been up for a few months, and that the splunk+data was rsynced over several times and tested. ...
See more...
Thanks for the replies. Please note that there was some planning in doing this, that the new system had been up for a few months, and that the splunk+data was rsynced over several times and tested. We obviously failed on the testing aspect, I have a single contact for users of this application, and relied on their reports of everything looking OK. Our system is only used weekdays, so after the Saturday change and realization that things were amiss, resulted in turning off the new server and reverting to the original server, so no unplanned production outage, system is running as it was before. I have to say (from inventsekars' reply) that in step 3, it is not immediately clear to me that I am to reinstall splunk on top of my original /opt/splunk directory rsynced over from the existing system. I would have thought that all the existing config files would be overwritten, and it would then seem to be like a new install. That is why I installed 8.2.6 splunk on the new server first, and then rsynced the existing data on top of it, to ensure any configuration files were intact. I am opening a ticket with Splunk today, to go over the process and investigate why the data was not reachable by the application. I appreciate the suggestions, but I do not know how to "check _internal index for errors" or do "tstats". Maybe I should request my employer send me for Splunk admin training, if they are expecting my to administer it. Cheers, Michael.