Security
Highlighted

Event not detected by indexer on [splunktcp-ssl] port

Explorer

I configured splunk to ingest logs on port 9338 with SSL enabled.

TCP dump on the port shows log data being received, but when I search on the indexer, this event is not captured

sudo tcpdump -i any port 9338

21 packets captured
42 packets received by filter
0 packets dropped by kernel

the log file in my /opt/splunk/var/log/splunk/splunkd.log

01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - IPv4 port 9336 is reserved for raw input
01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - IPv4 port 9336 will negotiate s2s protocol level 6
01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - IPv4 port 9337 is reserved for raw input
01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - IPv4 port 9337 will negotiate s2s protocol level 6
01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - IPv4 port 9338 is reserved for raw input
01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - IPv4 port 9338 will negotiate s2s protocol level 6
01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - Creating FwdDataSSLConfig SSL context. Will open port=IPv4 port 9338 with compression=1
01-23-2020 06:26:37.520 +0000 INFO  TcpInputConfig - IPv4 port 9338 is reserved for splunk 2 splunk (SSL)
01-23-2020 06:26:37.520 +0000 INFO  TcpInputConfig - IPv4 port 9338 will negotiate s2s protocol level 6
01-23-2020 06:26:38.343 +0000 WARN  HttpListener - Socket error from 127.0.0.1:44420 while idling: error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request

Please how can I fix this?

0 Karma
Highlighted

Re: Event not detected by indexer on [splunktcp-ssl] port

Explorer

Additional Log

01-23-2020 07:35:13.079 +0000 INFO  IndexWriter - openDatabases complete currentId=-1 idx=summary
01-23-2020 07:35:13.079 +0000 INFO  IndexProcessor - Initializing indexes took usec=700 reloading=false indexes_initialized=9
01-23-2020 07:35:13.087 +0000 INFO  TcpInputProc - Registering metrics callback for: tcpin_connections
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9336 is reserved for raw input
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9336 will negotiate s2s protocol level 6
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9337 is reserved for raw input
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9337 will negotiate s2s protocol level 6
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9338 is reserved for raw input
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9338 will negotiate s2s protocol level 6
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - Creating FwdDataSSLConfig SSL context. Will open port=IPv4 port 9338 with compression=1
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9338 is reserved for splunk 2 splunk (SSL)
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9338 will negotiate s2s protocol level 6
01-23-2020 07:35:13.088 +0000 INFO  TcpInputProc - Creating raw Acceptor for IPv4 port 9336 with Non-SSL
01-23-2020 07:35:13.088 +0000 INFO  TcpInputProc - Creating raw Acceptor for IPv4 port 9337 with Non-SSL
01-23-2020 07:35:13.088 +0000 INFO  TcpInputProc - Creating fwd data Acceptor for IPv4 port 9338 with SSL
01-23-2020 07:35:13.090 +0000 INFO  ExecProcessor - New scheduled exec process: /opt/splunk/bin/splunkd instrument-resource-usage
01-23-2020 07:35:13.090 +0000 INFO  ExecProcessor -     interval: 0 ms
01-23-2020 07:35:13.090 +0000 INFO  ExecProcessor - interval="0 * * * *" is a valid cron schedule
0 Karma
Highlighted

Re: Event not detected by indexer on [splunktcp-ssl] port

Explorer

Input.conf

[splunktcp-ssl://:9338]
disabled = 0

[SSL]
serverCert = /opt/splunk/etc/auth/mycerts/myNewServerCertificate.pem
sslPassword = $7$0XwJ4Q3QxJXxrDeuKBzS3XYyHaNhoBc05xBbcnkb1miQVcrEFlEfEixMjTv3arI99g==
requireClientCert = false
0 Karma
Highlighted

Re: Event not detected by indexer on [splunktcp-ssl] port

SplunkTrust
SplunkTrust

Have you configured intermediateCA or rootCA in server.conf on Indexer ? (Ref doc. https://docs.splunk.com/Documentation/Splunk/8.0.1/Security/ConfigureSplunkforwardingtousesignedcert...)

0 Karma
Highlighted

Re: Event not detected by indexer on [splunktcp-ssl] port

Explorer

No, I have not done this yet. I will do it and update this thread if it solves the problem. Thanks

0 Karma
Highlighted

Re: Event not detected by indexer on [splunktcp-ssl] port

Explorer

This was solved by changing the

 [splunktcp-ssl://:9338]
 disabled = 0

to

 [tcp-ssl://:9338]
 disabled = 0

splunktcp-ssl should be used if the forwarder is a Splunk forwarder. But in my case I was using Splunk's Java logging library to forwards logs to the Splunk instance.

I have added a gist here (https://gist.github.com/OluwoleOyetoke/b38aecbb47323ad25840d711cef8bf1) to help others who may need to do the same in the future

View solution in original post

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.