Universal forwarders no longer sending data - SSL23 unknown protocol

[adamsimpsondevo]

Our universal forwarders can no longer connect to the indexer, seemingly after upgrading openssl to the newest version due to the heartbleed vulnerability.

Telneting from UF clients to the indexer works fine.

On the clients we get this error message:
TcpOutputProc - Found a candidate indexer which is currently not connected. xxx.xxx.xxx.xxx:9997, client refCount=0, client=NULL

On the indexer we see this error message at the same time:
ERROR TcpInputProc - Error encountered for connection from src=xxx.xxx.xxx.xxx:48990. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

[MuS · Accepted solution]

Hi adamsimpsondevops,

Splunk ships with it's own openssl package and does not rely on the OS install of openssl. You can verify the Splunk openssl version by using the following command:

$SPLUNK_HOME/bin/splunk cmd openssl version -a

which returns something like this:

OpenSSL 1.0.1e-fips 11 Feb 2013
built on: Tue Oct 29 22:05:26 PDT 2013
platform: linux-x86_64
options:  bn(64,64) rc4(16x,int) des(idx,cisc,16,int) blowfish(idx)
compiler: gcc -static-libgcc -L/opt/splunk/lib/static-libstdc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -static-libgcc -Wa,--noexecstack -Wa,--noexecstack -m64 -DL_ENDIAN -DTERMIO -O3 -Wall -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -I/usr/local/ssl/fips-2.0/include -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
OPENSSLDIR: "/opt/splunk/openssl"

As you can see I did not update my development box yet 😉

Back to your question: did you relink the openssl libraries on the forwarder OS so the UF will use the OS openssl or is it still using Splunk's integrated openssl? You can verify it by comparing the outputs of the command above and a openssl version -a on the cli of your OS.

cheers, MuS

[ekcsoc]

Hi,

I am facing similar issue (same error), I have bunch of Universal Forwarder (version 6.5.2, 8.0.3) running in FIPS Mode. So I re-installed the UF on these linux servers. 

Still I am getting the same error on my heavy forwarder running 7.2.7. 

Could you please specifically mention, if I need to enable FIPS on Heavy Forwarder also. I also didn't see any document giving complete detail that where all FIPS should be enabled. If you could route me to correct document would be great.