We are in the midst of migrating our Splunk from Ubuntu to RHEL 9 Stigged. Supposedly, this should have been copy paste, minor clean up, start Splunk. it has been anything but that. Error after error and problem after problem. I finally have it narrowed down to a certificate error; an error that doesn't really make sense since everything worked fine on the other server. The server name and IP address are the same as the old server. They weren't set that way initially since I was using rsync to move things but after a couple errors, I realized I needed to fix that. I set the IP and hostname to the same as the old one, powered off the old server, and rebooted the new one to make sure settings were correct.  Currently, I'm seeing a bunch of python errors in splunkd.log.    Error ExecProcessor -message from "<splunkhome>/bin/python3.7 /<splunkhome>/etc/apps/search/bin/quarantine_files.py" from splunk.quarantine_files.configs import get_all_configs.   This error reports dozens of times but changes the file from. Eventually it ends with   ERROR ExecProcessor [33034 ExecProcessor] - message from "/<splunkhome>/bin/python3.7 /<splunkhome>/etc/apps/search/bin/quarantine_files.py" MemoryError IndexProcessor [32848 MainThread] - handleSignal : Disabling streaming searches. IndexProcessor [32848 MainThread] - request state change from=RUN to=SHUTDOWN_SIGNALED ProcessRunner [32850 ProcessRunner] - Unexpected EOF from process runner child! ProcessRunner [32850 ProcessRunner] -helper process seems to have died (child killed by signal 15: Terminated) !     In mongod.log I get "error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: unsupported certificate purpose. Ending connection from <server IP>". When running splunk start, it says waiting for web server..... and eventually fails saying web interface does not seem to be available. I'm unclear if this or the python errors causes Splunk to stop but within about 10 minutes of starting Splunk, it stops. Previous to this, it was giving me an error about the kvstore (I don't remember it). After some digging, I discovered that I also had to add a section to the server.conf for kvstore to use a certificate. That's apparently a requirement when running FIPS... Except another RHEL system runs it fine without the certificate... The next error was about the hostname not matching the cert. The hostname it listed was 127.0.0.1 which is not what the hostname is set to. I manually set it to the server IP with SPLUNK_BINDIP in splunk-launch.conf and that cleared that issue but it still doesn't load the web page and I now get the certificate error. It feels like it's connecting to itself as a client for some reason and failing the certificate. Did I configure something wrong? We've never had to set that manually so I found it odd that we to when moving to RHEL. I found another post that indicated I could check if the certificate was a server and client with the -purpose. Unfortunately (and unsurprisingly) it's server only. I have been trying to figure out either A) how to get it to stop asking for the client certificate or B) how to create a certificate that acts as both server and client... or I guess C) how to create the client certificate and where to place it. All of our certs are probably server only. We have our own CA so we aren't doing self-signed. Any thoughts? Any tips on any of these issues would be appreciated. ... View more

I have an odd task I'm trying to fulfill and I'm not entirely sure how to go about it.  We have a print server that forwards logs to Splunk. We also have multiple printers that are on a separate VLAN that only the print server can see. The objective is to see if we can pull the logs directly from the printer and forward them to Splunk. From what I've been reading, this should be possible by setting up the print server as a sort of intermediate forwarder? I believe the process is to have the printers redirect their logs to the print server to a specific folder, then add that folder to the list of logs being reported in the Splunk forwarder. Does that sound correct? Has anyone done this before? Any instructions that could make this easier? I'm fairly new to Splunk and I'm still learning how to set things up so as many details as possible would be helpful.   Thanks. ... View more

To start with, I am very new to Splunk and I've been stumbling my way through this with varying degrees of success.  We recently upgraded Splunk from 8.2 to 9.1.2. We noticed the new SSL requirements but went we have a self-signed cert but the website shows as not secure. We wanted to make sure everything was as secure as possible. We created an actual CA Cert chain and redirected the web.conf to the cert along with the key. I had issues with this at first because we weren't using a passphrase on the cert creation but we fixed that and it seems to accept it. Now the webpage seems to load, but it takes an incredibly long time. Once loaded, we should be able to login with LDAP. That's no longer working. I tried the local admin and it thinks for a while and then goes to a "Oops. The server encountered an unexpected condition which prevented it from fulfilling the request. Click here to return to Splunk homepage." page.  This is on the deploy server.  I changed the server.conf to use the cert as well though that doesn't appear to make a difference. I checked the openldap.conf and added the cert to that but then the page wouldn't load anymore. (doing a splunk restart between each change).  I'm not sure which logs to even look at to find the problem. I have gone through the documentation to setup the TLS which we want to do for interserver communication and for the webpage. the forwarders aren't necessary right now. Can anyone give me a clue what I might be doing wrong? EDIT: I did discover this error in the splunkd.log relating to my cert. Only post I've found so far says to combine the key and pem into a single file it can use. message="error:0906D06C:PEM routines:PEM_read_bio:no start line Here's my config files server.conf       [general] serverName = servername.com [changed for privacy reason] pass4SymmKey =[redacted] [sslConfig] # turns on TLS certificate host name validation sslVerifyServerName = true serverCert = /opt/splunk/etc/auth/servername.com.pem #sslPassword =[redacted] #SSL No longer valid option # sslPassword = [redacted] # turns on TLS certificate host name validation cliVerifyServerName = true sslPassword = [redacted] # Reference the file that contains all root certificate authority certificates combined together sslRootCAPath = /opt/splunk/etc/auth/servername.com.pem sslCommonNameList = servername.com, servername [pythonSslClientConfig] #sslVerifyServerCert = true #sslVerifyServerName = true [lmpool:auto_generated_pool_download-trial] description = auto_generated_pool_download-trial quota = MAX slaves = * stack_id = download-trial [lmpool:auto_generated_pool_forwarder] description = auto_generated_pool_forwarder quota = MAX slaves = * stack_id = forwarder [lmpool:auto_generated_pool_free] description = auto_generated_pool_free quota = MAX slaves = * stack_id = free [lmpool:auto_generated_pool_enterprise] description = auto_generated_pool_enterprise quota = MAX slaves = * stack_id = enterprise [license] active_group = Enterprise [kvstore] storageEngineMigration = true         web.conf       [settings] enableSplunkWebSSL = true privKeyPath = /opt/splunk/etc/auth/myprivate.key serverCert = /opt/splunk/etc/auth/servername.com.pem sslPassword =[redacted]         authentication.conf [authentication] authSettings = ldapserver.com authType = LDAP [roleMap_ldapserver.com] admin = SplunkAdmins [ldapserver.com] SSLEnabled = 1 anonymous_referrals = 1 bindDN = CN=ServiceAccount,CN=AccountFolder,DC=SubOrg,DC=Org,DC=com bindDNpassword = [redacted] charset = utf8 emailAttribute = mail enableRangeRetrieval = 0 groupBaseDN = OU=Groups,OU=Users & Computers,OU=MainFolder,DC=SubOrg,DC=Org,DC=com groupMappingAttribute = dn groupMemberAttribute = member groupNameAttribute = cn host = ldapserver.SubOrg.Org.Com nestedGroups = 0 network_timeout = 20 pagelimit = -1 port = 636 realNameAttribute = displayname sizelimit = 1000 timelimit = 15 userBaseDN = OU=Users,OU=Users & Computers,OU=MainFolder,DC=SubOrg,DC=Org,DC= com userNameAttribute = samaccountname   ldap.conf # See ldap.conf(5) for details # This file should be world readable but not world writable. ssl start_tls TLS_REQCERT demand TLS_CACERT /opt/splunk/etc/auth/ldapserver.pem # The following provides modern TLS configuration that guarantees forward- # secrecy and efficiency. This configuration drops support for old operating # systems (Windows Server 2008 R2 and earlier). # To add support for Windows Server 2008 R2 set TLS_PROTOCOL_MIN to 3.1 and # add these ciphers to TLS_CIPHER_SUITE: # ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA: # ECDHE-RSA-AES128-SHA # TLS_PROTOCOL_MIN: 3.1 for TLSv1.0, 3.2 for TLSv1.1, 3.3 for TLSv1.2. TLS_PROTOCOL_MIN 3.3 TLS_CIPHER_SUITE ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256> #TLS_CACERT absolute path to trusted certificate of LDAP server. For example /opt/splunk/etc/openldap/certs/mycertificate.pem #TLS_CACERTDIR absolute path to directory that contains trusted certificates of LDAP server. For example /opt/splunk/etc/openldap/certs   ... View more