Searching _internal for source=sc4s shows:
srlssydr01 syslog-ng 174 - [meta sequenceId="32595295"] Message(s) dropped while sending message to destination; driver='d_hec_fmt#0', worker_index='5', time_reopen='10', batch_size='19'
and
srlssydr01 syslog-ng 174 - [meta sequenceId="32594764"] http: handled by response_action; action='drop', url='https://http-inputs-acme.splunkcloud.com:443/services/collector/event', status_code='400', driver='d_hec_fmt#0', location='root generator dest_hec:5:5'
This can happen when you're trying to send to an index that doesn't exist. You can confirm this by turning on logging to d_hec_debug in /opt/sc4s/env_file and looking at the index-named folder list in /opt/sc4s/archive/debug and confirming all of those indexes exist.
Create the index or update splunk_metadata.csv to change the destination index for a given source key.
Thanks mbonsack in the sc4s community slack channel for the guidance. Posting here for visibility/googling.
This can happen when you're trying to send to an index that doesn't exist. You can confirm this by turning on logging to d_hec_debug in /opt/sc4s/env_file and looking at the index-named folder list in /opt/sc4s/archive/debug and confirming all of those indexes exist.
Create the index or update splunk_metadata.csv to change the destination index for a given source key.
Thanks mbonsack in the sc4s community slack channel for the guidance. Posting here for visibility/googling.
I am getting this all of the time and A the index exists and i can test it with curl and when sc4s starts it shows it is able to connect - it is annoying. what else can i check it is not well documented.
errors
- - syslog-ng 149 - [meta sequenceId="100"]Server disconnected while preparing messages for sending, trying again; driver='d_hec_fmt_other#0', location='root generator dest_hec:5:5', worker_index='3', time_reopen='10', batch_size='2'
| ||
1/21/25 2:41:42.705 PM | - - syslog-ng 149 - [meta sequenceId="100"]http: error sending HTTP request; url='https://somehost.com:3001/services/collector/event', error='Failed sending data to the peer', worker_index='3', driver='d_hec_fmt_other#0', location='root generator dest_hec:5:5'
|
It is clean at startup
SC4S_ENV_CHECK_HEC: Splunk HEC connection test successful to index=sddc_internal for sourcetype=sc4s:fallback...
SC4S_ENV_CHECK_HEC: Splunk HEC connection test successful to index=sddc_internal for sourcetype=sc4s:events...
syslog-ng checking config
sc4s version=3.34.1
Configuring the health check port to: 8080
[2025-01-21 13:36:54 +0000] [135] [INFO] Starting gunicorn 23.0.0
[2025-01-21 13:36:54 +0000] [135] [INFO] Listening at: http://0.0.0.0:8080 (135)
[2025-01-21 13:36:54 +0000] [135] [INFO] Using worker: sync
[2025-01-21 13:36:54 +0000] [138] [INFO] Booting worker with pid: 138
starting syslog-ng