Splunk Search

Universal forwarders no longer sending data - SSL23 unknown protocol

adamsimpsondevo
Engager

Our universal forwarders can no longer connect to the indexer, seemingly after upgrading openssl to the newest version due to the heartbleed vulnerability.

Telneting from UF clients to the indexer works fine.

On the clients we get this error message:
TcpOutputProc - Found a candidate indexer which is currently not connected. xxx.xxx.xxx.xxx:9997, client refCount=0, client=NULL

On the indexer we see this error message at the same time:
ERROR TcpInputProc - Error encountered for connection from src=xxx.xxx.xxx.xxx:48990. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

Tags (2)
1 Solution

MuS
SplunkTrust
SplunkTrust

Hi adamsimpsondevops,

Splunk ships with it's own openssl package and does not rely on the OS install of openssl. You can verify the Splunk openssl version by using the following command:

$SPLUNK_HOME/bin/splunk cmd openssl version -a

which returns something like this:

OpenSSL 1.0.1e-fips 11 Feb 2013
built on: Tue Oct 29 22:05:26 PDT 2013
platform: linux-x86_64
options:  bn(64,64) rc4(16x,int) des(idx,cisc,16,int) blowfish(idx)
compiler: gcc -static-libgcc -L/opt/splunk/lib/static-libstdc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -static-libgcc -Wa,--noexecstack -Wa,--noexecstack -m64 -DL_ENDIAN -DTERMIO -O3 -Wall -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -I/usr/local/ssl/fips-2.0/include -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
OPENSSLDIR: "/opt/splunk/openssl"

As you can see I did not update my development box yet 😉

Back to your question: did you relink the openssl libraries on the forwarder OS so the UF will use the OS openssl or is it still using Splunk's integrated openssl? You can verify it by comparing the outputs of the command above and a openssl version -a on the cli of your OS.

cheers, MuS

View solution in original post

MuS
SplunkTrust
SplunkTrust

Hi adamsimpsondevops,

Splunk ships with it's own openssl package and does not rely on the OS install of openssl. You can verify the Splunk openssl version by using the following command:

$SPLUNK_HOME/bin/splunk cmd openssl version -a

which returns something like this:

OpenSSL 1.0.1e-fips 11 Feb 2013
built on: Tue Oct 29 22:05:26 PDT 2013
platform: linux-x86_64
options:  bn(64,64) rc4(16x,int) des(idx,cisc,16,int) blowfish(idx)
compiler: gcc -static-libgcc -L/opt/splunk/lib/static-libstdc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -static-libgcc -Wa,--noexecstack -Wa,--noexecstack -m64 -DL_ENDIAN -DTERMIO -O3 -Wall -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -I/usr/local/ssl/fips-2.0/include -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
OPENSSLDIR: "/opt/splunk/openssl"

As you can see I did not update my development box yet 😉

Back to your question: did you relink the openssl libraries on the forwarder OS so the UF will use the OS openssl or is it still using Splunk's integrated openssl? You can verify it by comparing the outputs of the command above and a openssl version -a on the cli of your OS.

cheers, MuS

ekcsoc
Path Finder

Hi,

 

I am facing similar issue (same error), I have bunch of Universal Forwarder (version 6.5.2, 8.0.3) running in FIPS Mode. So I re-installed the UF on these linux servers. 

 

Still I am getting the same error on my heavy forwarder running 7.2.7. 

 

Could you please specifically mention, if I need to enable FIPS on Heavy Forwarder also. I also didn't see any document giving complete detail that where all FIPS should be enabled. If you could route me to correct document would be great.

 

 

Tags (2)
0 Karma
Get Updates on the Splunk Community!

What’s new on Splunk Lantern in August

This month’s Splunk Lantern update gives you the low-down on all of the articles we’ve published over the past ...

Welcome to the Future of Data Search & Exploration

You have more data coming at you than ever before. Over the next five years, the total amount of digital data ...

This Week's Community Digest - Splunk Community Happenings [8.3.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...