Splunk Search

Universal forwarders no longer sending data - SSL23 unknown protocol

adamsimpsondevo
Engager

Our universal forwarders can no longer connect to the indexer, seemingly after upgrading openssl to the newest version due to the heartbleed vulnerability.

Telneting from UF clients to the indexer works fine.

On the clients we get this error message:
TcpOutputProc - Found a candidate indexer which is currently not connected. xxx.xxx.xxx.xxx:9997, client refCount=0, client=NULL

On the indexer we see this error message at the same time:
ERROR TcpInputProc - Error encountered for connection from src=xxx.xxx.xxx.xxx:48990. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

Tags (2)
1 Solution

MuS
SplunkTrust
SplunkTrust

Hi adamsimpsondevops,

Splunk ships with it's own openssl package and does not rely on the OS install of openssl. You can verify the Splunk openssl version by using the following command:

$SPLUNK_HOME/bin/splunk cmd openssl version -a

which returns something like this:

OpenSSL 1.0.1e-fips 11 Feb 2013
built on: Tue Oct 29 22:05:26 PDT 2013
platform: linux-x86_64
options:  bn(64,64) rc4(16x,int) des(idx,cisc,16,int) blowfish(idx)
compiler: gcc -static-libgcc -L/opt/splunk/lib/static-libstdc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -static-libgcc -Wa,--noexecstack -Wa,--noexecstack -m64 -DL_ENDIAN -DTERMIO -O3 -Wall -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -I/usr/local/ssl/fips-2.0/include -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
OPENSSLDIR: "/opt/splunk/openssl"

As you can see I did not update my development box yet 😉

Back to your question: did you relink the openssl libraries on the forwarder OS so the UF will use the OS openssl or is it still using Splunk's integrated openssl? You can verify it by comparing the outputs of the command above and a openssl version -a on the cli of your OS.

cheers, MuS

View solution in original post

MuS
SplunkTrust
SplunkTrust

Hi adamsimpsondevops,

Splunk ships with it's own openssl package and does not rely on the OS install of openssl. You can verify the Splunk openssl version by using the following command:

$SPLUNK_HOME/bin/splunk cmd openssl version -a

which returns something like this:

OpenSSL 1.0.1e-fips 11 Feb 2013
built on: Tue Oct 29 22:05:26 PDT 2013
platform: linux-x86_64
options:  bn(64,64) rc4(16x,int) des(idx,cisc,16,int) blowfish(idx)
compiler: gcc -static-libgcc -L/opt/splunk/lib/static-libstdc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -static-libgcc -Wa,--noexecstack -Wa,--noexecstack -m64 -DL_ENDIAN -DTERMIO -O3 -Wall -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -I/usr/local/ssl/fips-2.0/include -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
OPENSSLDIR: "/opt/splunk/openssl"

As you can see I did not update my development box yet 😉

Back to your question: did you relink the openssl libraries on the forwarder OS so the UF will use the OS openssl or is it still using Splunk's integrated openssl? You can verify it by comparing the outputs of the command above and a openssl version -a on the cli of your OS.

cheers, MuS

ekcsoc
Path Finder

Hi,

 

I am facing similar issue (same error), I have bunch of Universal Forwarder (version 6.5.2, 8.0.3) running in FIPS Mode. So I re-installed the UF on these linux servers. 

 

Still I am getting the same error on my heavy forwarder running 7.2.7. 

 

Could you please specifically mention, if I need to enable FIPS on Heavy Forwarder also. I also didn't see any document giving complete detail that where all FIPS should be enabled. If you could route me to correct document would be great.

 

 

Tags (2)
0 Karma
Get Updates on the Splunk Community!

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...

Platform Highlights | January 2023 Newsletter

 January 2023Peace on Earth and Peace of Mind With Business ResilienceAll organizations can start the new year ...