How to fix SplunkDB "Unsupported or unrecognized SSL message"?

[emallinger]

Hi everyone,

I'm struggling with SplunkDB connect and HEC.

I have a monoinstance splunk that has all roles. I have multiple UF and use deployment server with HEC.

I'm currently trying to use SplunkDB connect on that instance to read some DB data and write it in an index and I keep having the error message : "Unsupported or unrecognized SSL message".

I checked the port (8088), seems fine. SSL is enabled on HEC and splunkd. (default parameters)

I wonder if it is possible that splunk db connect uses HEC entry on localhost:8088 if I have deported the HEC entry on the UFs with "useDeploymentServer".

Could it explain the ssl error ?

I tried to use dbx_settings.conf but it does not seems to use my second entry :

[hec]
maxRetryWhenHecUnavailable = 3
hecUris = localhost:8088,splunk-hec.qualgend:8088

Logs sample :

127.0.0.1 - - [03/nov./2022:15:30:08 +0000] "GET /api/taskserver HTTP/1.1" 200 414 "-" "python-requests/2.25.0" 2
	
2022-11-03 16:30:00.286 +0100  [Scheduled-Job-Executor-4] DEBUG c.s.d.s.dbinput.recordreader.DbInputRecordReader - action=closing_db_reader task=rising_mcipe_qualif
	
2022-11-03 16:30:00.286 +0100 INFO  c.s.dbx.server.task.listeners.JobMetricsListener - action=collect_job_metrics connection=mcipe_qualif jdbc_url=null db_read_time=0 hec_record_process_time=3 format_hec_success_count=69 status=FAILED input_name=rising_mcipe_qualif batch_size=1000 error_threshold=N/A is_jmx_monitoring=false start_time=2022-11-03_04:30:00 end_time=2022-11-03_04:30:00 duration=21 read_count=69 write_count=0 error_count=0
	
2022-11-03 16:30:00.285 +0100  [Scheduled-Job-Executor-4] INFO  org.easybatch.core.job.BatchJob - Job 'rising_mcipe_qualif' finished with status: FAILED
	
2022-11-03 16:30:00.285 +0100  [Scheduled-Job-Executor-4] ERROR org.easybatch.core.job.BatchJob - Unable to write records
javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
	at java.base/sun.security.ssl.SSLSocketInputRecord.handleUnknownRecord(SSLSocketInputRecord.java:451)
	at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:175)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
Afficher toutes les 35 lignes
	
2022-11-03 16:30:00.285 +0100  [Scheduled-Job-Executor-4] ERROR c.s.d.s.dbinput.recordwriter.CheckpointUpdater - action=skip_checkpoint_update_batch_writing_failed
javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
	at java.base/sun.security.ssl.SSLSocketInputRecord.handleUnknownRecord(SSLSocketInputRecord.java:451)
	at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:175)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
Afficher toutes les 35 lignes
	
2022-11-03 16:30:00.285 +0100  [Scheduled-Job-Executor-4] ERROR c.s.d.s.task.listeners.RecordWriterMetricsListener - action=unable_to_write_batch
javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
	at java.base/sun.security.ssl.SSLSocketInputRecord.handleUnknownRecord(SSLSocketInputRecord.java:451)
	at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:175)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
Afficher toutes les 35 lignes
	
2022-11-03 16:30:00.279 +0100  [Scheduled-Job-Executor-4] INFO  c.s.d.s.dbinput.recordwriter.HttpEventCollector - action=writing_events_via_http_event_collector record_count=69
	
2022-11-03 16:30:00.279 +0100  [Scheduled-Job-Executor-4] INFO  c.s.d.s.dbinput.recordwriter.HttpEventCollector - action=writing_events_via_http_event_collector
	
2022-11-03 16:30:00.279 +0100  [Scheduled-Job-Executor-4] INFO  c.s.dbx.server.dbinput.recordwriter.HecEventWriter - action=write_records batch_size=69

Do you have any idea what I did wrong ? Any clue would be greatly appreciated !

Thanks in advance,

Ema

[emallinger · Accepted solution]

Hi everyone,

With the help of a newly found splunker friend ;-), the solution is here :

https://vbrainstorm.com/splunk-blog-series-setting-up-hec-on-multiple-splunk-heavy-forwarders/

Work around is :

- do not use SplunkDBConnect as well as HEC with DS on the same machine (monoinstance case)

OR

- If you have to :

   1/ use a copy of splunk_httpinput app in deployment-apps for DS to use

   2/ Untick "use DS" on the HEC general settings

=> DB Connect will now work just fine

Thanks !

Ema