I'm attempting to setup splunk enterprise in a docker container using the official splunk image. I have been unsuccessful in getting ssl from the forwarders to the indexer configured. As far as I can tell, my certs are fine. I've tried enabling further debugging, but none of the errors seem to point me in the right direction.
indexer splunkd.log
07-15-2017 07:41:52.160 +0000 INFO TcpInputConfig - IPv4 port 9997 is reserved for splunk 2 splunk (SSL)
07-15-2017 07:41:52.161 +0000 INFO TcpInputProc - Creating raw Acceptor for IPv4 port 1514 with Non-SSL
07-15-2017 07:41:52.161 +0000 INFO TcpInputProc - Creating fwd data Acceptor for IPv4 port 9997 with SSL
07-15-2017 07:43:45.404 +0000 ERROR TcpInputProc - Error encountered for connection from src=10.101.21.34:36346. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
07-15-2017 07:43:45.413 +0000 ERROR TcpInputProc - Error encountered for connection from src=10.101.21.34:36348. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
indexer inputs.conf
#[default]
#host = splunkenterprise
[splunktcp-ssl:9997]
disabled=0
[SSL]
serverCert = /opt/splunk/etc/auth/splunk.cert.chain.pem
indexer inputs.conf
[general]
serverName = splunkenterprise
pass4SymmKey = $1$liNoIVdm5xPP
[sslConfig]
sslRootCAPath = /opt/splunk/etc/auth/intermediate-chain.cert.pem
sslPassword = $1$wW88fRIgrEHP
[lmpool:auto_generated_pool_download-trial]
description = auto_generated_pool_download-trial
quota = MAX
slaves = *
stack_id = download-trial
[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder
[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free
forwarder outputs.conf
[tcpout:group1]
server=splunk.mydomain.net:9997
disabled = 0
[tcpout:splunkssl]
sslRootCAPath = /etc/pki/ca-trust/source/anchors/intermediate-chain.cert.pem
forwarder server.conf
sslConfig]
sslRootCAPath = /etc/pki/ca-trust/source/anchors/intermediate-chain.cert.pem
sslPassword = $1$rCb/2hSpZ34D
[general]
pass4SymmKey = $1$+2qrhlHvLCwD
[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder
[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free
/splunk cmd openssl s_client -connect splunk.mydomain.net:9997
New, TLSv1/SSLv3, Cipher is AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : AES256-GCM-SHA384
Session-ID: 443457B09EEBEE91F8B72DE5132E970CCDBD14D96A1BF5BE02FE34ED6EA631D9
Session-ID-ctx:
Master-Key: B1D47C8F8EDD71D957E7BBC78B946B3FCFC4B6FB5B5527C3E16C6ADDE7C1DF7A6B950E8B2DC148EFCA4A70D88BC6035E
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - d6 cc 25 7b f8 a9 96 eb-70 16 9c ce 01 99 7f 0e ..%{....p.......
0010 - fd 7d 06 ec 4f cc 9f 63-27 00 b4 c2 19 b7 fc c0 .}..O..c'.......
0020 - e9 c5 0c 9c 2c 21 ed df-28 34 bd 4f 00 68 87 d2 ....,!..(4.O.h..
0030 - b2 cc bb 2a ff 32 51 26-dc 4f 7d d8 36 94 ea 96 ...*.2Q&.O}.6...
0040 - 74 d6 16 16 35 1e 3e c9-84 6c f0 c1 4a 3a 8d 7c t...5.>..l..J:.|
0050 - 1b 8d 67 bf 77 95 a2 1a-4a 4b df 2e 8a 12 4d fb ..g.w...JK....M.
0060 - 05 e5 39 74 28 2a 5a 35-39 70 57 09 f0 5c ac aa ..9t(*Z59pW..\..
0070 - 31 82 b1 42 24 8f 80 6e-6b a9 97 c5 31 2d e9 6a 1..B$..nk...1-.j
0080 - c1 03 25 c0 27 53 90 b0-7d 2e 1a 1a a5 24 73 6c ..%.'S..}....$sl
0090 - 89 4e 3d f7 2d f7 dd 1c-ce 0c 65 36 f6 27 55 11 .N=.-.....e6.'U.
Compression: 1 (zlib compression)
Start Time: 1500105568
Timeout : 300 (sec)
Verify return code: 19 (self signed certificate in certificate chain)
the server and ssl settings should be in the same tcpout stanza.
[tcpout:group1]
server=splunk.mydomain.net:9997
clientCert = /etc/pki/ca-trust/source/anchors/intermediate-chain.cert.pem
sslPassword= clientCertPasswordIfApplicable
the server and ssl settings should be in the same tcpout stanza.
[tcpout:group1]
server=splunk.mydomain.net:9997
clientCert = /etc/pki/ca-trust/source/anchors/intermediate-chain.cert.pem
sslPassword= clientCertPasswordIfApplicable
Switching the outputs stanza as you advised to
[tcpout:group1]
server=splunk.mydomain.net:9997
disabled = 0
sslRootCAPath = /etc/pki/ca-trust/source/anchors/intermediate-chain.cert.pem
clientCert = /opt/splunkforwarder/etc/auth/splunk.mydomain.net.cert.pair.pem
worked. The client is now logging
7-15-2017 18:39:55.889 +0000 INFO TcpOutputProc - Connected to idx=10.101.21.34:9997
The indexer is logging
07-15-2017 18:34:59.115 +0000 DEBUG TcpInputConfig - connection_host=ip for 10.101.21.34
It does seem odd that the instructions show that it us supposed to be configured differently
[tcpout:group1]
server=10.1.1.197:9997
disabled = 0
[tcpout:splunkssl]
useClientSSLCompression = <true> Disabling tls compression can cause bandwidth issues.
sslPassword = The password for the CAcert
sslCommonNameToCheck = (Optional) <commonName1>, <commonName2>, ... sslVerifyServerCert must be enabled to use common name checking. Defaults to no common name checking.
sslAltNameToCheck = (Optional) <alternateName1>, <alternateName2>, ... sslVerifyServerCert must be enabled to use common name checking. Defaults to no common name checking.
sslVerifyServerCert = Defaults to false. If true, you must make sure that the server you are connecting to can be authenticated to. When enabled, the common name and the alternate name of the server are checked.
cipherSuite = (Optional) Splunk uses any specified cipher string for the input processors. If not set, Splunk uses the default cipher string provided by OpenSSL.
Leave a Comment
at the bottom of the docs page telling them that the information is outdated/incorrect.
Since this "sslRootCAPath" is deprecated, you don't need it. I would also be interested in seeing which instructions specified those settings as I don't see it in outputs.conf documentation.
I submitted feedback. Thanks!
If you can share a link,
I can ask the author to revise.
Did you recently upgrade to 6.6.x?
Do you have some older UFs?
Are you using SSL on the UFs to communicate?
If so, it sounds like the V2 vs V3 SSL problem that you can find several answers for here.
This is a new install of the indexer and forwarder. In my original configuration I was running 6.5.3 on the indexer and 6.6.0 for the forwarder.
You did get me thinking. I was running indexer 6.5.3 and a 6.6.0 forwarder. I did bring the forwarder down to version 6.5.3, but I'm seeing the same errors.