Event not detected by indexer on [splunktcp-ssl] port

[oluwoleoyetoke]

I configured splunk to ingest logs on port 9338 with SSL enabled.

TCP dump on the port shows log data being received, but when I search on the indexer, this event is not captured

sudo tcpdump -i any port 9338

21 packets captured
42 packets received by filter
0 packets dropped by kernel

the log file in my /opt/splunk/var/log/splunk/splunkd.log

01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - IPv4 port 9336 is reserved for raw input
01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - IPv4 port 9336 will negotiate s2s protocol level 6
01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - IPv4 port 9337 is reserved for raw input
01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - IPv4 port 9337 will negotiate s2s protocol level 6
01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - IPv4 port 9338 is reserved for raw input
01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - IPv4 port 9338 will negotiate s2s protocol level 6
01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - Creating FwdDataSSLConfig SSL context. Will open port=IPv4 port 9338 with compression=1
01-23-2020 06:26:37.520 +0000 INFO  TcpInputConfig - IPv4 port 9338 is reserved for splunk 2 splunk (SSL)
01-23-2020 06:26:37.520 +0000 INFO  TcpInputConfig - IPv4 port 9338 will negotiate s2s protocol level 6
01-23-2020 06:26:38.343 +0000 WARN  HttpListener - Socket error from 127.0.0.1:44420 while idling: error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request

Please how can I fix this?

[oluwoleoyetoke · Accepted solution]

This was solved by changing the

 [splunktcp-ssl://:9338]
 disabled = 0

to

 [tcp-ssl://:9338]
 disabled = 0

splunktcp-ssl should be used if the forwarder is a Splunk forwarder. But in my case I was using Splunk's Java logging library to forwards logs to the Splunk instance.

I have added a gist here (https://gist.github.com/OluwoleOyetoke/b38aecbb47323ad25840d711cef8bf1) to help others who may need to do the same in the future

[oluwoleoyetoke]

Additional Log

01-23-2020 07:35:13.079 +0000 INFO  IndexWriter - openDatabases complete currentId=-1 idx=summary
01-23-2020 07:35:13.079 +0000 INFO  IndexProcessor - Initializing indexes took usec=700 reloading=false indexes_initialized=9
01-23-2020 07:35:13.087 +0000 INFO  TcpInputProc - Registering metrics callback for: tcpin_connections
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9336 is reserved for raw input
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9336 will negotiate s2s protocol level 6
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9337 is reserved for raw input
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9337 will negotiate s2s protocol level 6
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9338 is reserved for raw input
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9338 will negotiate s2s protocol level 6
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - Creating FwdDataSSLConfig SSL context. Will open port=IPv4 port 9338 with compression=1
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9338 is reserved for splunk 2 splunk (SSL)
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9338 will negotiate s2s protocol level 6
01-23-2020 07:35:13.088 +0000 INFO  TcpInputProc - Creating raw Acceptor for IPv4 port 9336 with Non-SSL
01-23-2020 07:35:13.088 +0000 INFO  TcpInputProc - Creating raw Acceptor for IPv4 port 9337 with Non-SSL
01-23-2020 07:35:13.088 +0000 INFO  TcpInputProc - Creating fwd data Acceptor for IPv4 port 9338 with SSL
01-23-2020 07:35:13.090 +0000 INFO  ExecProcessor - New scheduled exec process: /opt/splunk/bin/splunkd instrument-resource-usage
01-23-2020 07:35:13.090 +0000 INFO  ExecProcessor -     interval: 0 ms
01-23-2020 07:35:13.090 +0000 INFO  ExecProcessor - interval="0 * * * *" is a valid cron schedule

[oluwoleoyetoke]

Input.conf

[splunktcp-ssl://:9338]
disabled = 0

[SSL]
serverCert = /opt/splunk/etc/auth/mycerts/myNewServerCertificate.pem
sslPassword = $7$0XwJ4Q3QxJXxrDeuKBzS3XYyHaNhoBc05xBbcnkb1miQVcrEFlEfEixMjTv3arI99g==
requireClientCert = false

[harsmarvania57]

Have you configured intermediateCA or rootCA in server.conf on Indexer ? (Ref doc. https://docs.splunk.com/Documentation/Splunk/8.0.1/Security/ConfigureSplunkforwardingtousesignedcert...)

[oluwoleoyetoke]

No, I have not done this yet. I will do it and update this thread if it solves the problem. Thanks