Splunk SSL communication between Splunk Universal Forwarder and Heavy forwarder- Experiencing error

[vikashumble]

Hello All,

I am stuck on one problem and I am not able to find the solution of it so far so need all your expertise to help me out.

My splunk setup which I have problem with: Splunk UF --> Splunk HF --> Splunk cloud

On splunk UF, I have a inputs configured to monitor a file. I am trying to configure SSL for data transfer between  Splunk UF and Splunk HF. I have placed Root CA and Server/Client certificate in SPLUNK_HOME/etc/certs directory. Below are my inputs (on HF) and outputs on (Splunk UF).

For sslRootCAPath path in inputs.conf and outputs.conf, I have been told by my client that even though the name is different (on HF and UF) but they are essentially same. 

Inputs.conf (on HF):

[splunktcp-ssl:9997]

#sslPassword = password

disabled = 0

requireClientCert = false

serverCert = /opt/splunk/etc/certs/Cert_HF.pem

sslRootCAPath = /opt/splunk/etc/certs/XXXX_Root_CA.pem

Outputs.conf (on UF):

[tcpout]

defaultGroup=spl_hfs

[tcpout:spl_hfs]

server = INDEXER_1:9997, INDEXER_2:9997

clientCert = C:\Program Files\SplunkUniversalForwarder\etc\certs\Cert_UF.pem

sslRootCAPath = C:\Program Files\SplunkUniversalForwarder\etc\certs\XXXX_Root.pem

#sslPassword = password

When I configure above settings and restart UF and HF, I see below error in HF Splunkd.log and none of the data (not even _internal from UF via HF) is indexed. I can see HF to Splunk cloud communication is working as expected. But my UF to HF is throwing below error.

Error: 

ERROR TcpInputProc [1899734 FwdDataReceiverThread] - Message rejected. Received unexpected message of size=369295616 bytes from src=XXXXXX:38998 in streaming mode. Maximum message size allowed=67108864. (::) Possible invalid source sending data to splunktcp port or valid source sending unsupported payload.

I have tried to look on google and see even read through splunk pages, tried configs/changes as suggested but I am still struggling to find a working solution for me. Any help in pointing me in right direction is highly appreciated.

Also, my few other questions are:

1. As my client mentioned that even though, root CA name on HF (XXXX_Root_CA.pem) and UF (XXXX_Root.pem) are different but they are same, is there any way/command using which, I can confirm that whether they are really same or different?
2. Does we need to have SAME root CA certificate distributed to HF and UFs for SSL communications or can they be different?
3. I have been told that there is no sslpassword attached with the certificates, is there any way/command I can confirm this myself rather than taking their word for it?
4. What else I can change/try in .conf files to see if this SSL config work?

Any replies on my issue is highly appreciated

Thanks

Vikas

[aasabatini]

Hi @vikashumble 

have you define the server.conf both the HF and UF?

please check the documentation

https://docs.splunk.com/Documentation/Splunk/8.2.6/Security/ConfigureSplunkforwardingtousesignedcert...

Also I think you have to create a folder to put your own certificates in this path:

/opt/splunk/etc/auth/

[vikashumble]

Hi @aasabatini 

Thanks for reply. Yes, server.conf has been configured with [sslConfig] stanza with sslrootCA attribute. I don't have screenshot of this handy else I would have attached it here. Even after that I am getting the same error again and again. 😞

Thanks, Vikas