SC4S to Splunk Cloud forwarding receiving errors?

[jell0r]

Hi,

I have a network background and know a little bit syslog and using the Splunk interface. This is however the first time I'm installing a new Splunk instance.

I've started up my Splunk cloud trial and spun up SC4S via podman (version 2.39.0) on RHEL 8.2.

It seems to be running fine, SC4S messages arriving in Splunk Cloud

- - syslog-ng 149 - [meta sequenceId="1"]syslog-ng starting up; version='3.36.1'
host = splunk-sc4ssource = sc4ssourcetype = sc4s:events

I'm sending syslog from my test ASA to SC4S and with a tcpdump I can see it coming in:

07:46:13.658608 IP xxx.45.78.xxx.syslog > splunk-sc4s.internal.cloudapp.net.syslog: SYSLOG local7.debug, length: 101
07:46:13.735897 IP xxx.45.78.xxx.syslog > splunk-sc4s.internal.cloudapp.net.syslog: SYSLOG local7.info, length: 183
07:46:13.962147 IP xxx.45.78.xxx.syslog > splunk-sc4s.internal.cloudapp.net.syslog: SYSLOG local7.info, length: 155
07:46:16.550565 IP xxx.45.78.xxx.syslog > splunk-sc4s.internal.cloudapp.net.syslog: SYSLOG local7.info, length: 166

Problem is, in Splunk I'm only getting http errors which seems to come from the above syslog messages forwarded by SC4S:

- syslog-ng 149 - [meta sequenceId="18"]curl: error sending HTTP request; url='https://prd-p-xxxxxx.splunkcloud.com:8088/services/collector/event', error='Timeout was reached', worker_index='2', driver='d_hec_fmt#0', location='root generator dest_hec:5:5'
host = splunk-sc4ssource = sc4ssourcetype = sc4s:events 

Things I've configured on SC4S, only the basics:

/etc/sysctl.conf

net.core.rmem_default = 17039360
net.core.rmem_max = 17039360
net.ipv4.ip_forward = 1

/opt/sc4s/env_file

SC4S_DEST_SPLUNK_HEC_DEFAULT_URL=https://prd-p-xxxxx.splunkcloud.com:8088
SC4S_DEST_SPLUNK_HEC_DEFAULT_TOKEN=xxxxxxxx-9c3c-4918-8eb3-xxxxxxxxxxxxx
#Uncomment the following line if using untrusted SSL certificates 
SC4S_DEST_SPLUNK_HEC_DEFAULT_TLS_VERIFY=no

In splunk I've created the HEC token:

I've created the Indexes like this, looking at this everything seems to fall in the "main" index instead of "lastchangeindex", looks like that's not right, is it?

I must be missing something obvious here because this should be straightforward right?

Appreciate any input on this, thanks!

[mattymo]

So the issue you are having is the index it lands in correct?

It is likely that SC4S hec client is sending a default index. 

Hec clients settings in the payload override the settings you put on the token. Think of those as "if not set by the hec client". 

https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/FormateventsforHTTPEventCollector#Even...

Would check sc4s docs on setting indexes: https://splunk.github.io/splunk-connect-for-syslog/main/configuration/#log-path-overrides-of-index-o...

[RicoViq]

No replies, having same issue here.  It's something we messed up or isn't documented, just trying to figure out what.