After migration from Ubuntu to RHEL 9, certificate error

[Hiattech]

We are in the midst of migrating our Splunk from Ubuntu to RHEL 9 Stigged. Supposedly, this should have been copy paste, minor clean up, start Splunk. it has been anything but that. Error after error and problem after problem. I finally have it narrowed down to a certificate error; an error that doesn't really make sense since everything worked fine on the other server. The server name and IP address are the same as the old server. They weren't set that way initially since I was using rsync to move things but after a couple errors, I realized I needed to fix that. I set the IP and hostname to the same as the old one, powered off the old server, and rebooted the new one to make sure settings were correct. 

Currently, I'm seeing a bunch of python errors in splunkd.log. 

Error ExecProcessor -message from "<splunkhome>/bin/python3.7  /<splunkhome>/etc/apps/search/bin/quarantine_files.py" from splunk.quarantine_files.configs import get_all_configs.

This error reports dozens of times but changes the file from. Eventually it ends with

ERROR ExecProcessor [33034 ExecProcessor] - message from "/<splunkhome>/bin/python3.7  /<splunkhome>/etc/apps/search/bin/quarantine_files.py" MemoryError
IndexProcessor [32848 MainThread] - handleSignal : Disabling streaming searches.
IndexProcessor [32848 MainThread] - request state change from=RUN to=SHUTDOWN_SIGNALED
ProcessRunner [32850 ProcessRunner] - Unexpected EOF from process runner child!
ProcessRunner [32850 ProcessRunner] -helper process seems to have died (child killed by signal 15: Terminated) !

In mongod.log I get "error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: unsupported certificate purpose. Ending connection from <server IP>". When running splunk start, it says waiting for web server..... and eventually fails saying web interface does not seem to be available. I'm unclear if this or the python errors causes Splunk to stop but within about 10 minutes of starting Splunk, it stops.

Previous to this, it was giving me an error about the kvstore (I don't remember it). After some digging, I discovered that I also had to add a section to the server.conf for kvstore to use a certificate. That's apparently a requirement when running FIPS... Except another RHEL system runs it fine without the certificate...

The next error was about the hostname not matching the cert. The hostname it listed was 127.0.0.1 which is not what the hostname is set to. I manually set it to the server IP with SPLUNK_BINDIP in splunk-launch.conf and that cleared that issue but it still doesn't load the web page and I now get the certificate error. It feels like it's connecting to itself as a client for some reason and failing the certificate.

Did I configure something wrong? We've never had to set that manually so I found it odd that we to when moving to RHEL.

I found another post that indicated I could check if the certificate was a server and client with the -purpose. Unfortunately (and unsurprisingly) it's server only. I have been trying to figure out either A) how to get it to stop asking for the client certificate or B) how to create a certificate that acts as both server and client... or I guess C) how to create the client certificate and where to place it. All of our certs are probably server only. We have our own CA so we aren't doing self-signed. Any thoughts?

Any tips on any of these issues would be appreciated.