Why is syslog-ng dropping events sent to SC4S's destination d_hec_fmt?

[gf13579]

Searching _internal for source=sc4s shows:

srlssydr01 syslog-ng 174 - [meta sequenceId="32595295"] Message(s) dropped while sending message to destination; driver='d_hec_fmt#0', worker_index='5', time_reopen='10', batch_size='19'

and

srlssydr01 syslog-ng 174 - [meta sequenceId="32594764"] http: handled by response_action; action='drop', url='https://http-inputs-acme.splunkcloud.com:443/services/collector/event', status_code='400', driver='d_hec_fmt#0', location='root generator dest_hec:5:5'

[gf13579 · Accepted solution]

This can happen when you're trying to send to an index that doesn't exist. You can confirm this by turning on logging to d_hec_debug in /opt/sc4s/env_file and looking at the index-named folder list in /opt/sc4s/archive/debug and confirming all of those indexes exist.

Create the index or update splunk_metadata.csv to change the destination index for a given source key.

Thanks mbonsack in the sc4s community slack channel for the guidance. Posting here for visibility/googling.

[tigerdice]

I am getting this all of the time and A the index exists and i can test it with curl and when sc4s starts it shows it is able to connect - it is annoying.  what else can i check it is not well documented.

[tigerdice]

errors

- - syslog-ng 149 - [meta sequenceId="100"]Server disconnected while preparing messages for sending, trying again; driver='d_hec_fmt_other#0', location='root generator dest_hec:5:5', worker_index='3', time_reopen='10', batch_size='2' host = dev-ipz001-splunk05 source = sc4s sourcetype = sc4s:events
	1/21/25 2:41:42.705 PM	- - syslog-ng 149 - [meta sequenceId="100"]http: error sending HTTP request; url='https://somehost.com:3001/services/collector/event', error='Failed sending data to the peer', worker_index='3', driver='d_hec_fmt_other#0', location='root generator dest_hec:5:5' host = splunk05 source = sc4s sourcetype = sc4s:events

[tigerdice]

It is clean at startup

SC4S_ENV_CHECK_HEC: Splunk HEC connection test successful to index=sddc_internal for sourcetype=sc4s:fallback...
SC4S_ENV_CHECK_HEC: Splunk HEC connection test successful to index=sddc_internal for sourcetype=sc4s:events...
syslog-ng checking config
sc4s version=3.34.1
Configuring the health check port to: 8080
[2025-01-21 13:36:54 +0000] [135] [INFO] Starting gunicorn 23.0.0
[2025-01-21 13:36:54 +0000] [135] [INFO] Listening at: http://0.0.0.0:8080 (135)
[2025-01-21 13:36:54 +0000] [135] [INFO] Using worker: sync
[2025-01-21 13:36:54 +0000] [138] [INFO] Booting worker with pid: 138
starting syslog-ng