Failed to send log with syslog - syslog-ng[1]: curl: error sending HTTP request - 'Couldn\'t connect to server'

[sutom]

I am trying to run the splunk connect syslog via podman, here is the reference links -
https://splunk-connect-for-syslog.readthedocs.io/en/latest/gettingstarted/#offline-container-install...
https://splunk-connect-for-syslog.readthedocs.io/en/latest/gettingstarted/podman-systemd-general/

My podman container is up and running, all the configuration on place as per doc instructions - But I am facing a issue related to sending logs HTTP request. Below is my configuration file and activity logs.

My env_file

[root@hostname ~]# cat /opt/sc4s/env_file
SPLUNK_HEC_URL=https://http-singh-sudhir.splunkcloud.com:443
SPLUNK_HEC_TOKEN=Z93TSS87-F826-19V1-01W1-Q9Q8G1G8264
#Uncomment the following line if using untrusted SSL certificates
#SC4S_DEST_SPLUNK_HEC_DEFAULT_TLS_VERIFY=no
SC4S_DEST_SPLUNK_HEC_DEFAULT_DISKBUFF_DIR=/opt/sc4s/storage/volumes

Using above config the manual curl command is successful

[root@hostname ~]# curl -k https://http-singh-sudhir.splunkcloud.com:443/services/collector/event?channel=Q9Q8G1W5-Z93T-F826-19V1-Q9Q8G1G8264 -H "Authorization: Splunk Z93TSS87-F826-19V1-01W1-Q9Q8G1G8264 " -d '{"event": "hello_world"}'
{"text":"Success","code":0}[root@hostname ~]# ^C

But with same config, podman logs SC4S is throwing error

[root@hostname ~]# /usr/bin/podman logs SC4S
'/opt/syslog-ng/etc/conf.d/local/context/compliance_meta_by_source.conf.example' -> '/opt/syslog-ng/etc/conf.d/local/context/compliance_meta_by_source.conf'
'/opt/syslog-ng/etc/conf.d/local/context/compliance_meta_by_source.csv.example' -> '/opt/syslog-ng/etc/conf.d/local/context/compliance_meta_by_source.csv'
'/opt/syslog-ng/etc/conf.d/local/context/splunk_index.csv.example' -> '/opt/syslog-ng/etc/conf.d/local/context/splunk_index.csv'
'/opt/syslog-ng/etc/conf.d/local/context/vendor_product_by_source.conf.example' -> '/opt/syslog-ng/etc/conf.d/local/context/vendor_product_by_source.conf'
'/opt/syslog-ng/etc/conf.d/local/context/vendor_product_by_source.csv.example' -> '/opt/syslog-ng/etc/conf.d/local/context/vendor_product_by_source.csv'
'/opt/syslog-ng/etc/local_config/destinations/README.md' -> '/opt/syslog-ng/etc/conf.d/local/config/destinations/README.md'
'/opt/syslog-ng/etc/local_config/filters/README.md' -> '/opt/syslog-ng/etc/conf.d/local/config/filters/README.md'
'/opt/syslog-ng/etc/local_config/filters/example.conf' -> '/opt/syslog-ng/etc/conf.d/local/config/filters/example.conf'
'/opt/syslog-ng/etc/local_config/log_paths/README.md' -> '/opt/syslog-ng/etc/conf.d/local/config/log_paths/README.md'
'/opt/syslog-ng/etc/local_config/log_paths/lp-example.conf.tmpl' -> '/opt/syslog-ng/etc/conf.d/local/config/log_paths/lp-example.conf.tmpl'
'/opt/syslog-ng/etc/local_config/log_paths/lp-example.conf' -> '/opt/syslog-ng/etc/conf.d/local/config/log_paths/lp-example.conf'
'/opt/syslog-ng/etc/local_config/sources/README.md' -> '/opt/syslog-ng/etc/conf.d/local/config/sources/README.md'
syslog-ng checking config
sc4s version=v1.12.0
syslog-ng starting
Aug 16 11:44:12 hostname syslog-ng[1]: syslog-ng starting up; version='3.25.1'
Aug 16 11:44:12 hostname syslog-ng-config: sc4s version=v1.12.0
Aug 16 11:44:12 hostname syslog-ng[1]: curl: error sending HTTP request; url='https://http-singh-sudhir.splunkcloud.com:443/services/collector/event', error='Couldn\'t connect to server', worker_index='1', driver='d_hec_internal#0', location='/opt/syslog-ng/etc/conf.d/destinations/splunk_hec_internal.conf:2:5'
Aug 16 11:44:12 hostname syslog-ng[1]: Server disconnected while preparing messages for sending, trying again; driver='d_hec_internal#0', location='/opt/syslog-ng/etc/conf.d/destinations/splunk_hec_internal.conf:2:5', worker_index='1', time_reopen='10', batch_size='1'
Aug 16 11:44:12 hostname syslog-ng[1]: curl: error sending HTTP request; url='https://http-singh-sudhir.splunkcloud.com:443/services/collector/event', error='Couldn\'t connect to server', worker_index='0', driver='d_hec_internal#0', location='/opt/syslog-ng/etc/conf.d/destinations/splunk_hec_internal.conf:2:5'
Aug 16 11:44:12 hostname syslog-ng[1]: Server disconnected while preparing messages for sending, trying again; driver='d_hec_internal#0', location='/opt/syslog-ng/etc/conf.d/destinations/splunk_hec_internal.conf:2:5', worker_index='0', time_reopen='10', batch_size='1'

I am not able to understand what is missing here from my side. if is curl fails then it should be in both cases, looking forward to your help. please point out what is wrong with this.

[harsmarvania57]

Hi,

Looks like you are using very old version of SC4S, please use latest version.

[sutom]

Thanks for the hit @harsmarvania57 

As per suggestion I have updated the version and also manage the config according to new version, but again I am getting same kind of error.

update env_file.

[root@hostname ~]# cat /opt/sc4s/env_file
SC4S_DEST_SPLUNK_HEC_DEFAULT_URL=https://http-singh-sudhir.splunkcloud.com:443
SC4S_DEST_SPLUNK_HEC_DEFAULT_TOKEN=Z93TSS87-F826-19V1-01W1-Q9Q8G1G8264
#Uncomment the following line if using untrusted SSL certificates
#SC4S_DEST_SPLUNK_HEC_DEFAULT_TLS_VERIFY=no
SC4S_DEST_GLOBAL_ALTERNATES=d_hec_debug

podman logs SC4S

[root@hostname sc4s]# podman logs SC4S
curl: (7) Failed to connect to http-singh-sudhir.splunkcloud.com port 443: Connection timed out
SC4S_ENV_CHECK_HEC: Invalid Splunk HEC URL, invalid token, or other HEC connectivity issue index=main. sourcetype=sc4s:fallback
Startup will continue to prevent data loss if this is a transient failure.

syslog-ng checking config
sc4s version=1.86.4
starting goss
starting syslog-ng
Aug 16 16:07:35.327 hostname syslog-ng[166]: syslog-ng starting up; version='3.32.1'
Aug 16 16:07:36.700 hostname syslog-ng[166]: curl: error sending HTTP request; url='http-singh-sudhir.splunkcloud.com:443/services/collector/event', error='Couldn\'t connect to server', worker_index='2', driver='d_hec_fmt#0', location='root generator dest_hec:5:5'
Aug 16 16:07:36.700 hostname syslog-ng[166]: curl: error sending HTTP request; url='http-singh-sudhir.splunkcloud.com:443/services/collector/event', error='Couldn\'t connect to server', worker_index='3', driver='d_hec_fmt#0', location='root generator dest_hec:5:5'
Aug 16 16:07:36.700 hostname syslog-ng[166]: Server disconnected while preparing messages for sending, trying again; driver='d_hec_fmt#0', location='root generator dest_hec:5:5', worker_index='2', time_reopen='10', batch_size='198'
Aug 16 16:07:36.700 hostname syslog-ng[166]: Server disconnected while preparing messages for sending, trying again; driver='d_hec_fmt#0', location='root generator dest_hec:5:5', worker_index='3', time_reopen='10', batch_size='198'

Log from Debug file - 

[root@hostname sc4s_events]# cat 2021-08-16-hec.log
curl -k -u "sc4s HEC debug:$SC4S_DEST_SPLUNK_HEC_DEFAULT_TOKEN" "https://http-singh-sudhir.splunkcloud.com:443/services/collector/event" -d '{"time":"1629130055.327","sourcetype":"sc4s:events","source":"sc4s","index":"main","host":"sudhir4321","fields":{"sc4s_vendor_product":"sc4s_events","sc4s_syslog_facility":"syslog","sc4s_loghost":"sudhir4321","sc4s_container":"sudhir4321"},"event":"2021-08-16T16:07:35.327+00:00 sudhir4321 syslog-ng 166 - [meta sequenceId=\"1\"] syslog-ng starting up; version='3.32.1'"}'
curl -k -u "sc4s HEC debug:$SC4S_DEST_SPLUNK_HEC_DEFAULT_TOKEN" "https://http-singh-sudhir.splunkcloud.com:443/services/collector/event" -d '{"time":"1629130056.401","sourcetype":"sc4s:events:startup:out","source":"sc4s","index":"main","host":"sudhir4321","fields":{"sc4s_vendor_product":"sc4s_events","sc4s_syslog_facility":"user","sc4s_loghost":"sudhir4321","sc4s_container":"sudhir4321"},"event":"syslog-ng-config: sc4s version=1.86.4"}'
curl -k -u "sc4s HEC debug:$SC4S_DEST_SPLUNK_HEC_DEFAULT_TOKEN" "https://http-singh-sudhir.splunkcloud.com:443/services/collector/event" -d '{"time":"1629130056.700","sourcetype":"sc4s:events","source":"sc4s","index":"main","host":"sudhir4321","fields":{"sc4s_vendor_product":"sc4s_events","sc4s_syslog_facility":"syslog","sc4s_loghost":"sudhir4321","sc4s_container":"sudhir4321"},"event":"2021-08-16T16:07:36.700+00:00 sudhir4321 syslog-ng 166 - [meta sequenceId=\"3\"] curl: error sending HTTP request; url='https://http-singh-sudhir.splunkcloud.com:443/services/collector/event', error='Couldn\\'t connect to server', worker_index='2', driver='d_hec_fmt#0', location='root generator dest_hec:5:5'"}'

What could be the issue now, can you please help me to understand.

[s2_splunk]

I don't think your HEC URL is correct if you are targeting a SplunkCloud stack.

It should be 

https://http-inputs-<stackname>.splunkcloud.com

[sutom]

Thanks for response @s2_splunk 

I have sensitized the URL- my actual URL is-

"https://http-inputs-sudhir.splunkcloud.com:443"

[s2_splunk]

Are you saying you modified all the log entries you have posted above to obfuscate the actual HEC endpoint?

If that's the case, I would try to test your Cloud Stacks's HEC functionality with a curl command from that machine: 

curl -k  https://http-inputs-<stack>.splunkcloud.com:443/services/collector/event -H "Authorization: Splunk <valid_token>" -d '{"event": "hello world"}'

And see if that works. 

If it times out as well, something on your network doesn't allow the outbound connection.

[sutom]

@s2_splunkyou are right. I have modified the HEC URL.

Here is the curl command response -

[root@hostname ~]# curl -k  https://http-inputs-sudhir.splunkcloud.com:443/services/collector/event -H "Authorization: Splunk Z93TSS87-F826-19V1-01W1-Q9Q8G1G8264" -d '{"event": "hello world"}'
{"text":"Success","code":0}
[root@hostname ~]#

Output is success but with script is failing

[s2_splunk]

Sorry, didn't see that you had already tried that.

Do you have any proxy configuration on your server that your podman container may not "see"?

[s2_splunk]

Which version of podman are you running?

[sutom]

podman -v
podman version 3.0.2-dev

[s2_splunk]

Hmmm, not sure where to go from here. By default, podman should pick up system-wide proxy settings if this is really your issue.

On other thing you could try is to run the image in interactive mode and issue the curl command from within the container to see if that really is your issue.

You could also try explicitly setting your proxy variables in the env_file and see if that changes anything.

If you have a support entitlement, you can open a support case. Maybe others more familiar with podman and/or how to bypass your proxy for the node SC4S runs on have better ideas.

[sutom]

Thanks @s2_splunk

As you suggested, I tried to run the CURL command in normal mode as well as image interactive mode and it got failed in interactive mode.

Normal mode CURL command -

[root@hostname ~]# curl -k https://http-inputs-sudhir.splunkcloud.com:443/services/collector -H "Authorization: Splunk Z93TSS87-F826-19V1-01W1-Q9Q8G1G8264" -d '{​"event": "hello_world", "sourcetype":"mysourcetype"}​'
{​"text":"Success","code":0}​

CURL command in image interactive mode - it's timed out

podman exec -it containerid /bin/sh
sh-4.4# curl -k -v https://http-inputs-sudhir.splunkcloud.com:443/services/collector -H "Authorization: Splunk Z93TSS87-F826-19V1-01W1-Q9Q8G1G8264" -d '{​"event": "hello_world", "sourcetype":"mycontainer"}​'
* Trying 4.54.253.184...
* TCP_NODELAY set
* connect to 4.54.253.184 port 443 failed: Connection timed out
* Trying 4.54.142.7...
* TCP_NODELAY set
^C
sh-4.4# exit

Looks like podman is not picking up system-wide proxy settings, Any suggestion here how I can troubleshoot or force podman for system proxy

[s2_splunk · Accepted solution]

Can you do an nslookup on your splunkcloud.com hostname and confirm that the IP address is the same as the one reported in your connection timeout?

I haven't run across this before. Typically, podman (and docker) should pick up system-wide proxy settings without a problem.

You can try to add the proxy variables to the env_file of your SC4S container. Get the current HTTPS_PROXY environment variable setting from your host and copy/paste into env_file. Restart/Retry.

[sutom]

Thanks @s2_splunk 

I did nslookup for my splunkcloud.com hostname and got the same IP address as in my connection timeout?

$ nslookup http-inputs-sudhir.splunkcloud.com
Non-authoritative answer:
Server:  dns.google
Address:  8.8.8.8
Name:    sudhir-indexers-15287165932.xx-xxxxx-xx.elb.amazonaws.com
Addresses:  4.54.253.184
          4.54.142.7
Aliases:  http-inputs-sudhir.splunkcloud.com

Then I set the HTTPS_PROXY  = value(HTTPS_PROXY environment variable setting from my host) in env_file of your SC4S container and tried the CURL command in image interactive mode and it got

succeed - Thank you.

But again I got another issue related to CA certificate-

syslog-ng[165]: curl: error sending HTTP request; url='https://http-inputs-sudhir.splunkcloud.com:443/services/collector/event', error='Peer certificate cannot be authenticated with given CA certificates', worker_index='3', driver='d_hec_fmt#0', location='root generator dest_hec:5:5'

I have few  SSL certificates installed on machine, but note sure whether those are creating problem.

Need suggestion, do I need to uninstall my ssl certs from here or  need to install ssl root CA on cloud HEC-end point.  I am hoping now this will not trouble more as earlier.

[sutom]

Dear @s2_splunk ,

Finally I got my SC4S logs on the place,

Above error got resolve by setting the below variable-

SC4S_DEST_SPLUNK_HEC_TLS_VERIFY = no

[sutom]

@s2_splunkI have tried the curl command and the output is given above.

Yes, we have a proxy configuration on the server. could you please help me with some hits - like how I can identify which proxy configuration affecting podman container.