Why is syslog-ng dropping events sent to SC4S's destination d_hec_fmt?

[gf13579]

Searching _internal for source=sc4s shows:

srlssydr01 syslog-ng 174 - [meta sequenceId="32595295"] Message(s) dropped while sending message to destination; driver='d_hec_fmt#0', worker_index='5', time_reopen='10', batch_size='19'

and

srlssydr01 syslog-ng 174 - [meta sequenceId="32594764"] http: handled by response_action; action='drop', url='https://http-inputs-acme.splunkcloud.com:443/services/collector/event', status_code='400', driver='d_hec_fmt#0', location='root generator dest_hec:5:5'

[gf13579 · Accepted solution]

This can happen when you're trying to send to an index that doesn't exist. You can confirm this by turning on logging to d_hec_debug in /opt/sc4s/env_file and looking at the index-named folder list in /opt/sc4s/archive/debug and confirming all of those indexes exist.

Create the index or update splunk_metadata.csv to change the destination index for a given source key.

Thanks mbonsack in the sc4s community slack channel for the guidance. Posting here for visibility/googling.