I've been asked to assist another department with getting their Splunk configuration working with windows UFs. They have a single Linux-based 9.4.1 indexer that is successfully fed by a large number of Linux UFs. For the most part I haven't found anything really odd about it. They are using self-signed certs that have several years of validity left on them. FTR, I am not a windows admin so I am kind of grasping at straws here. Both their 'nix and windows UFs use Splunk's Deployment Server for configuration. All UFs are using the same fwd_to_loghost and ssl_bundle apps, the only difference is windowsconf_global or linux_global apps, as appropriate (I have verified the correct app is installed). They made an attempt a year or so to get this working, with no success. I believe I've removed all trace of it and have removed and reinstalled the UF (using 9.4.1 this time) on the windows host from scratch. The windows box connects to the Deployment Server and downloads the apps (fwd_to_loghost, ssl_bundle, and windowsconf_global) correctly but when it tries to connect to the indexer to send logs it fails. The indexer says: ERROR TcpInputProc [2957596 FwdDataReceiverThread-0] - Error encountered for connection from src=[redacted, correct IP address]:49902. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol The windows box has some interesting things to say in C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log: 04-24-2025 14:03:59.924 -0700 INFO TcpOutputProc [2948 parsing] - Initializing connection for non-ssl forwarding to loghost.biostat.washington.edu:9997 ... 04-24-2025 14:03:59.940 -0700 INFO CertificateData [2948 parsing] - channel=Forwarder, subject="emailAddress=[redacted],CN=loghost-uf.biostat.washington.edu,OU=Biostatistics,O=University of Washington,L=Seattle,ST=Washington,C=US", subjectAltName="DNS:keller-uf, DNS:keller-uf.biostat.washington.edu, DNS:loghost-uf, DNS:loghost-uf.biostat.washington.edu", serial=10, notValidBefore=1623099653, notValidAfter=1938459653, issuer="/C=US/ST=Washington/L=Seattle/O=UW/OU=Biostatistics/CN=zwickel.biostat.washington.edu/emailAddress=bite@uw.edu", sha256-fingerprint=10:31:07:BF:21:F2:49:41:34:E4:53:7F:89:C0:CB:81:99:6E:16:00:29:3E:C4:BC:C3:88:A1:CC:92:D0:AD:32 ... 04-24-2025 14:04:00.362 -0700 WARN X509Verify [5944 HTTPDispatch] - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see: <http://docs.splunk.com/Documentation/Splunk/latest/Security/Howtoself-signcertificates> 04-24-2025 14:04:00.381 -0700 INFO CertificateData [5944 HTTPDispatch] - channel=HTTPServer, subject="O=SplunkUser,CN=SplunkServerDefaultCert", subjectAltName="", serial=9814D004673F8828, notValidBefore=1745011134, notValidAfter=1839619134, issuer="/C=US/ST=CA/L=San Francisco/O=Splunk/CN=SplunkCommonCA/emailAddress=support@splunk.com", sha256-fingerprint=DC:75:CA:ED:54:2A:28:12:D4:A1:B9:DC:37:29:75:F4:9B:56:1F:A2:C7:33:BB:EB:EF:02:37:AC:6E:81:E4:CA I am not seeing anything in the log before the non-ssl line that appears to be an error, though it is a noisy log so it is quite possible I missed something. I have my working splunk configuration with functional Windows and Linux UFs that I am trying to base this work on. It does not have the non-ssl or SplunkServerDefaultCert log entries. I presume both are Bad Signs<tm>. Both my working system and this one have sslRootCAPath set in deployment-apps/fwd_to_loghost/default/outputs.conf: [tcpout] defaultGroup = splunkssl [tcpout:splunkssl] compressed = true server = loghost.biostat.washington.edu:9997 clientCert = $SPLUNK_HOME/etc/apps/ssl_bundle/default/UF/loghost-uf-bundle.crt sslPassword = [redacted] sslRootCAPath = $SPLUNK_HOME/etc/apps/ssl_bundle/default/biostat-ca.crt sslVerifyServerCert = true neither of them [had] sslRootCAPath set anywhere else in deployment-apps. I've tried adding a deployment-apps/windowsconf_global/default/server.conf, though ConfigureSplunkforwardingtousesignedcertificates seems to say this is only needed for non-windows hosts: [sslConfig] sslRootCAPath = $SPLUNK_HOME/etc/apps/ssl_bundle/default/biostat-ca.crt but the "unknown protocol" errors and non-ssl and SplunkServerDefaultCert log entries persist. As I said, I'm not a windows admin but given the windows hosts in the working environment are fine with paths like "$SPLUNK_HOME/etc/apps/ssl_bundle/default/..." in outputs.conf and there is a reference to a clearly self-signed cert in the log I have to presume these path entries are valid and working so it should be finding both the cert and the bundle. I've looked at the output of btool server & btool outputs, comparing it with the working instance, and I don't see any obvious or glaring problems. The new server.conf entry shows up in the output of btool server list so it is being seen but not having any impact on the problem. I presume the "unknown protocol" is because the windows UF is trying to use a non-ssl connection, per the UF's log file entry. I've read (and re-read, and re-re-read) https://docs.splunk.com/Documentation/Splunk/9.4.1/Security/ConfigureSplunkforwardingtousesignedcertificates and several forum posts that seem to be about this kind of problem but so far nothing seems to have addressed it. I have to try not to break the linux UFs that are working so I have to be careful what files I touch in deployment-apps - I'm trying to limit myself to only modifying things in windowsconf_global when possible. Where should I look to try to resolve this problem? Given the Linux UFs are working fine I presume the problem is somewhere in the config for the Windows UF. Thanks in advance for any assistance. ... View more