Security

Event not detected by indexer on [splunktcp-ssl] port

oluwoleoyetoke
Explorer

I configured splunk to ingest logs on port 9338 with SSL enabled.

TCP dump on the port shows log data being received, but when I search on the indexer, this event is not captured

sudo tcpdump -i any port 9338

21 packets captured
42 packets received by filter
0 packets dropped by kernel

the log file in my /opt/splunk/var/log/splunk/splunkd.log

01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - IPv4 port 9336 is reserved for raw input
01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - IPv4 port 9336 will negotiate s2s protocol level 6
01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - IPv4 port 9337 is reserved for raw input
01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - IPv4 port 9337 will negotiate s2s protocol level 6
01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - IPv4 port 9338 is reserved for raw input
01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - IPv4 port 9338 will negotiate s2s protocol level 6
01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - Creating FwdDataSSLConfig SSL context. Will open port=IPv4 port 9338 with compression=1
01-23-2020 06:26:37.520 +0000 INFO  TcpInputConfig - IPv4 port 9338 is reserved for splunk 2 splunk (SSL)
01-23-2020 06:26:37.520 +0000 INFO  TcpInputConfig - IPv4 port 9338 will negotiate s2s protocol level 6
01-23-2020 06:26:38.343 +0000 WARN  HttpListener - Socket error from 127.0.0.1:44420 while idling: error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request

Please how can I fix this?

0 Karma
1 Solution

oluwoleoyetoke
Explorer

This was solved by changing the

 [splunktcp-ssl://:9338]
 disabled = 0

to

 [tcp-ssl://:9338]
 disabled = 0

splunktcp-ssl should be used if the forwarder is a Splunk forwarder. But in my case I was using Splunk's Java logging library to forwards logs to the Splunk instance.

I have added a gist here (https://gist.github.com/OluwoleOyetoke/b38aecbb47323ad25840d711cef8bf1) to help others who may need to do the same in the future

View solution in original post

0 Karma

oluwoleoyetoke
Explorer

This was solved by changing the

 [splunktcp-ssl://:9338]
 disabled = 0

to

 [tcp-ssl://:9338]
 disabled = 0

splunktcp-ssl should be used if the forwarder is a Splunk forwarder. But in my case I was using Splunk's Java logging library to forwards logs to the Splunk instance.

I have added a gist here (https://gist.github.com/OluwoleOyetoke/b38aecbb47323ad25840d711cef8bf1) to help others who may need to do the same in the future

0 Karma

oluwoleoyetoke
Explorer

Additional Log

01-23-2020 07:35:13.079 +0000 INFO  IndexWriter - openDatabases complete currentId=-1 idx=summary
01-23-2020 07:35:13.079 +0000 INFO  IndexProcessor - Initializing indexes took usec=700 reloading=false indexes_initialized=9
01-23-2020 07:35:13.087 +0000 INFO  TcpInputProc - Registering metrics callback for: tcpin_connections
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9336 is reserved for raw input
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9336 will negotiate s2s protocol level 6
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9337 is reserved for raw input
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9337 will negotiate s2s protocol level 6
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9338 is reserved for raw input
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9338 will negotiate s2s protocol level 6
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - Creating FwdDataSSLConfig SSL context. Will open port=IPv4 port 9338 with compression=1
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9338 is reserved for splunk 2 splunk (SSL)
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9338 will negotiate s2s protocol level 6
01-23-2020 07:35:13.088 +0000 INFO  TcpInputProc - Creating raw Acceptor for IPv4 port 9336 with Non-SSL
01-23-2020 07:35:13.088 +0000 INFO  TcpInputProc - Creating raw Acceptor for IPv4 port 9337 with Non-SSL
01-23-2020 07:35:13.088 +0000 INFO  TcpInputProc - Creating fwd data Acceptor for IPv4 port 9338 with SSL
01-23-2020 07:35:13.090 +0000 INFO  ExecProcessor - New scheduled exec process: /opt/splunk/bin/splunkd instrument-resource-usage
01-23-2020 07:35:13.090 +0000 INFO  ExecProcessor -     interval: 0 ms
01-23-2020 07:35:13.090 +0000 INFO  ExecProcessor - interval="0 * * * *" is a valid cron schedule
0 Karma

oluwoleoyetoke
Explorer

Input.conf

[splunktcp-ssl://:9338]
disabled = 0

[SSL]
serverCert = /opt/splunk/etc/auth/mycerts/myNewServerCertificate.pem
sslPassword = $7$0XwJ4Q3QxJXxrDeuKBzS3XYyHaNhoBc05xBbcnkb1miQVcrEFlEfEixMjTv3arI99g==
requireClientCert = false
0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Have you configured intermediateCA or rootCA in server.conf on Indexer ? (Ref doc. https://docs.splunk.com/Documentation/Splunk/8.0.1/Security/ConfigureSplunkforwardingtousesignedcert...)

0 Karma

oluwoleoyetoke
Explorer

No, I have not done this yet. I will do it and update this thread if it solves the problem. Thanks

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...