Getting Data In

Why is syslog-ng dropping events sent to SC4S's destination d_hec_fmt?

gf13579
Communicator

Searching _internal for source=sc4s shows:

 

 

 

srlssydr01 syslog-ng 174 - [meta sequenceId="32595295"] Message(s) dropped while sending message to destination; driver='d_hec_fmt#0', worker_index='5', time_reopen='10', batch_size='19'

 

 

 

and

 

 

 

srlssydr01 syslog-ng 174 - [meta sequenceId="32594764"] http: handled by response_action; action='drop', url='https://http-inputs-acme.splunkcloud.com:443/services/collector/event', status_code='400', driver='d_hec_fmt#0', location='root generator dest_hec:5:5'

 

 

 

Labels (2)
0 Karma
1 Solution

gf13579
Communicator

This can happen when you're trying to send to an index that doesn't exist. You can confirm this by turning on logging to d_hec_debug in /opt/sc4s/env_file and looking at the index-named folder list in /opt/sc4s/archive/debug and confirming all of those indexes exist.

Create the index or update splunk_metadata.csv to change the destination index for a given source key.

Thanks mbonsack in the sc4s community slack channel for the guidance. Posting here for visibility/googling.

View solution in original post

gf13579
Communicator

This can happen when you're trying to send to an index that doesn't exist. You can confirm this by turning on logging to d_hec_debug in /opt/sc4s/env_file and looking at the index-named folder list in /opt/sc4s/archive/debug and confirming all of those indexes exist.

Create the index or update splunk_metadata.csv to change the destination index for a given source key.

Thanks mbonsack in the sc4s community slack channel for the guidance. Posting here for visibility/googling.

tigerdice
Loves-to-Learn

I am getting this all of the time and A the index exists and i can test it with curl and when sc4s starts it shows it is able to connect - it is annoying.  what else can i check it is not well documented.

0 Karma

tigerdice
Loves-to-Learn

errors

- - syslog-ng 149 - [meta sequenceId="100"]Server disconnected while preparing messages for sending, trying again; driver='d_hec_fmt_other#0', location='root generator dest_hec:5:5', worker_index='3', time_reopen='10', batch_size='2'
 1/21/25
2:41:42.705 PM
 
- - syslog-ng 149 - [meta sequenceId="100"]http: error sending HTTP request; url='https://somehost.com:3001/services/collector/event', error='Failed sending data to the peer', worker_index='3', driver='d_hec_fmt_other#0', location='root generator dest_hec:5:5'
0 Karma

tigerdice
Loves-to-Learn

It is clean at startup

 

SC4S_ENV_CHECK_HEC: Splunk HEC connection test successful to index=sddc_internal for sourcetype=sc4s:fallback...
SC4S_ENV_CHECK_HEC: Splunk HEC connection test successful to index=sddc_internal for sourcetype=sc4s:events...
syslog-ng checking config
sc4s version=3.34.1
Configuring the health check port to: 8080
[2025-01-21 13:36:54 +0000] [135] [INFO] Starting gunicorn 23.0.0
[2025-01-21 13:36:54 +0000] [135] [INFO] Listening at: http://0.0.0.0:8080 (135)
[2025-01-21 13:36:54 +0000] [135] [INFO] Using worker: sync
[2025-01-21 13:36:54 +0000] [138] [INFO] Booting worker with pid: 138
starting syslog-ng

0 Karma
Get Updates on the Splunk Community!

.conf25 Registration is OPEN!

Ready. Set. Splunk! Your favorite Splunk user event is back and better than ever. Get ready for more technical ...

Detecting Cross-Channel Fraud with Splunk

This article is the final installment in our three-part series exploring fraud detection techniques using ...

Splunk at Cisco Live 2025: Learning, Innovation, and a Little Bit of Mr. Brightside

Pack your bags (and maybe your dancing shoes)—Cisco Live is heading to San Diego, June 8–12, 2025, and Splunk ...