SC4S to Splunk Cloud forwarding receiving errors?

[jell0r]

Hi,

I have a network background and know a little bit syslog and using the Splunk interface. This is however the first time I'm installing a new Splunk instance.

I've started up my Splunk cloud trial and spun up SC4S via podman (version 2.39.0) on RHEL 8.2.

It seems to be running fine, SC4S messages arriving in Splunk Cloud

- - syslog-ng 149 - [meta sequenceId="1"]syslog-ng starting up; version='3.36.1'
host = splunk-sc4ssource = sc4ssourcetype = sc4s:events

I'm sending syslog from my test ASA to SC4S and with a tcpdump I can see it coming in:

07:46:13.658608 IP xxx.45.78.xxx.syslog > splunk-sc4s.internal.cloudapp.net.syslog: SYSLOG local7.debug, length: 101
07:46:13.735897 IP xxx.45.78.xxx.syslog > splunk-sc4s.internal.cloudapp.net.syslog: SYSLOG local7.info, length: 183
07:46:13.962147 IP xxx.45.78.xxx.syslog > splunk-sc4s.internal.cloudapp.net.syslog: SYSLOG local7.info, length: 155
07:46:16.550565 IP xxx.45.78.xxx.syslog > splunk-sc4s.internal.cloudapp.net.syslog: SYSLOG local7.info, length: 166

Problem is, in Splunk I'm only getting http errors which seems to come from the above syslog messages forwarded by SC4S:

- syslog-ng 149 - [meta sequenceId="18"]curl: error sending HTTP request; url='https://prd-p-xxxxxx.splunkcloud.com:8088/services/collector/event', error='Timeout was reached', worker_index='2', driver='d_hec_fmt#0', location='root generator dest_hec:5:5'
host = splunk-sc4ssource = sc4ssourcetype = sc4s:events 

Things I've configured on SC4S, only the basics:

/etc/sysctl.conf

net.core.rmem_default = 17039360
net.core.rmem_max = 17039360
net.ipv4.ip_forward = 1

/opt/sc4s/env_file

SC4S_DEST_SPLUNK_HEC_DEFAULT_URL=https://prd-p-xxxxx.splunkcloud.com:8088
SC4S_DEST_SPLUNK_HEC_DEFAULT_TOKEN=xxxxxxxx-9c3c-4918-8eb3-xxxxxxxxxxxxx
#Uncomment the following line if using untrusted SSL certificates 
SC4S_DEST_SPLUNK_HEC_DEFAULT_TLS_VERIFY=no

In splunk I've created the HEC token:

I've created the Indexes like this, looking at this everything seems to fall in the "main" index instead of "lastchangeindex", looks like that's not right, is it?

I must be missing something obvious here because this should be straightforward right?

Appreciate any input on this, thanks!

[mattymo]

So the issue you are having is the index it lands in correct?

It is likely that SC4S hec client is sending a default index. 

Hec clients settings in the payload override the settings you put on the token. Think of those as "if not set by the hec client". 

https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/FormateventsforHTTPEventCollector#Even...

Would check sc4s docs on setting indexes: https://splunk.github.io/splunk-connect-for-syslog/main/configuration/#log-path-overrides-of-index-o...

[RicoViq]

No replies, having same issue here.  It's something we messed up or isn't documented, just trying to figure out what.

[tigerdice]

same problem.  The index is correct

# docker logs -f sc4s
SC4S_ENV_CHECK_HEC: Splunk HEC connection test successful to index=sddc_internal for sourcetype=sc4s:fallback...
SC4S_ENV_CHECK_HEC: Splunk HEC connection test successful to index=sddc_internal for sourcetype=sc4s:events...
syslog-ng checking config
sc4s version=3.34.1
Configuring the health check port to: 8080
[2025-01-21 13:54:30 +0000] [129] [INFO] Starting gunicorn 23.0.0
[2025-01-21 13:54:30 +0000] [129] [INFO] Listening at: http://0.0.0.0:8080 (129)
[2025-01-21 13:54:30 +0000] [129] [INFO] Using worker: sync
[2025-01-21 13:54:30 +0000] [138] [INFO] Booting worker with pid: 138
starting syslog-ng  no errors on startup but still these sc4s:events keep coming no idea what they are and the are annoying.  The index is correct.

[mattymo]

Are you also on a cloud trial?

These could just be momentary server busy etc, be sure to check splunk internal logs  index=_internal source=*splunkd.log httpinputdatahandler)  to see if the payload hit a 503 or something then retried. It is "expected" that hec clients have to handle backpressure  or timeouts, so from time to time you may see a failed send, but as long as retry is successful, its "normal" unless you up your indexing layer to handle more traffic uninterrupted. 

The error says "timeout reached" so it could be that Splunk was to busy to answer (especially in standalone trail or small test boxes). 

Also please confirm the HEC full URL you are using. I believe you need to put the full URL

https://http-inputs.foo.splunkcloud.com/services/collector/event (or trial equivalient)

OP looks like they configured to just the cloud url on 8088, which is not a correct url for HEC. 

[tigerdice]

Here is the full env_file

SC4S_SOURCE_SYSLOG_PORTS=514,32514,32601,41514,41601,42514,42601

SC4S_DEST_SPLUNK_HEC_DEFAULT_URL=https://<>:3001/services/collector/event
SC4S_DEST_SPLUNK_HEC_DEFAULT_TOKEN=<>
SC4S_DEST_SPLUNK_HEC_DEFAULT_INDEX=<>
# SC4S_DEST_SPLUNK_HEC_DEFAULT_MODE=GLOBAL
# SC4S_DEST_SPLUNK_HEC_DEFAULT_FORMAT=json
# SC4S_DEST_SPLUNK_HEC_DEFAULT_TLS_VERIFY=no

SC4S_DEST_SPLUNK_HEC_TLS_CA_FILE=/etc/ssl/certs/ca-certificates.crt
SC4S_DEST_SPLUNK_HEC_TLS_CLIENT_CERT=/etc/syslog-ng/tls/splunk.crt
SC4S_DEST_SPLUNK_HEC_TLS_CLIENT_KEY=/etc/syslog-ng/tls/splunk.key

SC4S_SOURCE_TLS_ENABLE=yes
SC4S_SOURCE_TLS_KEY=/etc/syslog-ng/tls/server.key
SC4S_SOURCE_TLS_CERT=/etc/syslog-ng/tls/server.pem

SC4S_DEST_SPLUNK_HEC_DEFAULT_DISKBUFF_ENABLE=yes
SC4S_DEST_SPLUNK_HEC_DEFAULT_DISKBUFF_DIR=/var/lib/sc4s/disk-buffer

SC4S_HEALTH_CHECK_ENABLE=true
SC4S_LISTEN_INTERNAL_HEALTH_PORT=9129

SC4S_ETC=/etc/syslog-ng

SC4S_LISTEN_CHECKPOINT_SPLUNK_NOISE_CONTROL_SECONDS=30

SC4S_LISTEN_STATUS_PORT=9129
SC4S_LISTEN_DEFAULT_TCP_PORT=41514
SC4S_LISTEN_DEFAULT_UDP_PORT=42514
SC4S_LISTEN_DEFAULT_TLS_PORT=7514
SC4S_LISTEN_DEFAULT_RFC5426_PORT=41601
SC4S_LISTEN_DEFAULT_RFC6587_PORT=42601
SC4S_LISTEN_DEFAULT_RFC5425_PORT=7425

SC4S_HEALTH_CHECK_ENABLE=true

SC4S_DEST_SPLUNK_HEC_BATCH_SIZE=1
SC4S_DEST_SPLUNK_HEC_RETRY_LIMIT=1
SC4S_DEST_SPLUNK_HEC_RETRY_INTERVAL=5

SOURCE_ALL_SET=DEFAULT_TCP,DEFAULT_UDP

# SC4S_SEND_METRICS_TERMINAL=no

SC4S_DEBUG=false
SC4S_LOG_LEVEL=false
SC4S_DEFAULT_TIMEZONE=Europe/Berlin
PYTHONPATH=/var/lib/python-venv/lib/python3.12/site-packages:/etc/syslog-ng/python:/etc/syslog-ng/pylib
# Tunning settings
SC4S_DEST_SPLUNK_HEC_TIME_REOPEN=30
SC4S_DEST_SPLUNK_HEC_BATCH_LINES=100
SC4S_DEST_SPLUNK_HEC_BATCH_TIMEOUT=5000
SC4S_DEST_SPLUNK_HEC_KEEPALIVE=yes
SC4S_DEST_SPLUNK_HEC_WORKERS=8
SC4S_DEST_SPLUNK_HEC_DISKBUFF_ENABLE=yes
SC4S_DEST_SPLUNK_HEC_DISKBUFF_RELIABLE=yes
SC4S_DEST_SPLUNK_HEC_DISKBUFF_MEMBUFLENGTH=10000
SC4S_DEST_SPLUNK_HEC_DISKBUFF_DISKBUFSIZE=200000000

[tigerdice]

no it is not busy - it is getting basically a 400 error which means format if i take and only have the default destionation be d_hec_default or d_hec_other basically if i go only 2 one of the splunk HEC site it is fine.  It only gets the busy or format error on the second site if only the second site is configured so if i switch it whatever is the first site works and whatever is the second site doesnt work.  So no it is not busy or that it can not injest the logs it is sc4s doesnt seem to work well with multi destinations.  I am looking for help with someone that has done that.

[mattymo]

Where are you seeing the 400 error? The Hec client said timeout in this post, didnt seem to mention 400 bad request?

if its format then something is fundamentally wrong with the payload or you sending to the wrong url, etc. 

Anyways suggest you post your own post with any info and config, especially the hec url config and any splunk internal logs that align. 

Otherwise would try support or the github issues. 

found this issue that sounded kinda similar, but hard to tell without you providing config details or logs. 

https://github.com/splunk/splunk-connect-for-syslog/issues/1329