Create a directory to store all this. i.e. $SPLUNK_HOME/etc/auth/3rdpartycerts on the Indexer and Forwarders. Create a config file (3rdparty.cfg) that matches your third party CA. Keep changing the hostname or more the file around so you don't keep over righting it. Example: [ req ] default_bits = 2048 default_keyfile = hostname.key distinguished_name = req_distinguished_name [ req_distinguished_name ] 0.DC=DC=gov Press Enter 0.DC_default = gov 1.DC=DC=pc Press Enter 1.DC_default = pc 2.DC=DC=Microsoft Press Enter 2.DC_default = Microsoft 3.DC = Windows Domain 3.DC_default = mydomain commonName = Server Name commonName_max = 64 Create the key file for each indexer and forwarder. Recommend same bits as CA. # openssl genrsa -des3 -out hostname.key 2048 Create the request file for each indexer and forwarder using the config file: # openssl req -new -key eas01.key -out hostname.csr -config 3rdparty.cfg Paste the contents of the resulting hostname.csr file in your request to the CA. Download the resulting signed certificates and the CA certificate in pem (Base 64) format to your 3rdpartycerts directory on the Indexer and Forwarders. If any cert is in DER format, convert using the following: # openssl x509 -inform der -in cacert.crt -out cacert.pem Combine the server cert, server key and CA cert into a new server cert as follows: # cat hostname-cert.pem hostname.key cacert.pem > hostname.pem On the Indexer, ensure the following minimum entries exist in $SPLUNK_HOME/etc/system/local/inputs.conf file: [splunktcp-ssl:9996] compressed = true [SSL] password = $1$d9nAgrJsGkWc requireClientCert = false rootCA = $SPLUNK_HOME/etc/auth/3rdparty/cacert.pem serverCert = $SPLUNK_HOME/etc/auth/3rdparty/hostname.pem - Ensure the following minimum entries exist on the Forwarders $SPLUNK_HOME/etc/system/local/outputs.conf file: [tcpout] server = Indexer:9996 defaultGroup = splunkssl disabled = false [tcpout:splunkssl] compressed = true [tcpout-server://Indexer:9996] sslCertPath = $SPLUNK_HOME/etc/auth/3rdparty/hostname.pem sslPassword = $1$w6IdRdDtFjxG sslRootCAPath = $SPLUNK_HOME/etc/auth/3rdparty/cacert.pem Copy the third party CA cert to the /etc/pki/tls/certs directory on the Indexer and Forwarders. Create a hash link in /etc/pki/tls/certs directory so the third party CA cert will be trusted: # ln -s cacert.pem `openssl x509 -hash -noout -in cacert.pem`.0 Reboot the splunkd process on the Indexer and Forwarders and it should be working. ... View more

Reloaded Splunk software as a new install and the problem did not happen again. ... View more

I think I have answered my own question. After some research have determined that the Data files had been corrupted. So I removed them and restarted Splunk and now the SourcesTypes date is being populated. ... View more

The "over the wire" format for Splunk's communications between forwarders and indexers does use TCP as its transport. The "Splunk protocol" inside TCP is Splunk proprietary and (to my knowledge) not documented. If you are planning to build a receiver for Splunk forwarder data, be aware that Splunk can forward over a plain TCP socket to a 3rd party system. See http://www.splunk.com/base/Documentation/latest/Admin/Forwarddatatothird-partysystems ... View more

You may also want to check if the user role has the capability "rest_properties_get" as this error can occur if the default Splunk authentication is in use and this capability is missing. ... View more