topic Failed to send log with syslog - syslog-ng[1]: curl: error sending HTTP request - 'Couldn\'t connect to server' in Getting Data In

Failed to send log with syslog - syslog-ng[1]: curl: error sending HTTP request - 'Couldn\'t connect to server'

sutom — Mon, 16 Aug 2021 13:29:02 GMT

I am trying to run the splunk connect syslog via podman, here is the reference links -
https://splunk-connect-for-syslog.readthedocs.io/en/latest/gettingstarted/#offline-container-installation
https://splunk-connect-for-syslog.readthedocs.io/en/latest/gettingstarted/podman-systemd-general/

My podman container is up and running, all the configuration on place as per doc instructions - But I am facing a issue related to sending logs HTTP request. Below is my configuration file and activity logs.

My env_file

[root@hostname ~]# cat /opt/sc4s/env_file
SPLUNK_HEC_URL=https://http-singh-sudhir.splunkcloud.com:443
SPLUNK_HEC_TOKEN=Z93TSS87-F826-19V1-01W1-Q9Q8G1G8264
#Uncomment the following line if using untrusted SSL certificates
#SC4S_DEST_SPLUNK_HEC_DEFAULT_TLS_VERIFY=no
SC4S_DEST_SPLUNK_HEC_DEFAULT_DISKBUFF_DIR=/opt/sc4s/storage/volumes

Using above config the manual curl command is successful

[root@hostname ~]# curl -k https://http-singh-sudhir.splunkcloud.com:443/services/collector/event?channel=Q9Q8G1W5-Z93T-F826-19V1-Q9Q8G1G8264 -H "Authorization: Splunk Z93TSS87-F826-19V1-01W1-Q9Q8G1G8264 " -d '{"event": "hello_world"}'
{"text":"Success","code":0}[root@hostname ~]# ^C

But with same config, podman logs SC4S is throwing error

[root@hostname ~]# /usr/bin/podman logs SC4S
'/opt/syslog-ng/etc/conf.d/local/context/compliance_meta_by_source.conf.example' -> '/opt/syslog-ng/etc/conf.d/local/context/compliance_meta_by_source.conf'
'/opt/syslog-ng/etc/conf.d/local/context/compliance_meta_by_source.csv.example' -> '/opt/syslog-ng/etc/conf.d/local/context/compliance_meta_by_source.csv'
'/opt/syslog-ng/etc/conf.d/local/context/splunk_index.csv.example' -> '/opt/syslog-ng/etc/conf.d/local/context/splunk_index.csv'
'/opt/syslog-ng/etc/conf.d/local/context/vendor_product_by_source.conf.example' -> '/opt/syslog-ng/etc/conf.d/local/context/vendor_product_by_source.conf'
'/opt/syslog-ng/etc/conf.d/local/context/vendor_product_by_source.csv.example' -> '/opt/syslog-ng/etc/conf.d/local/context/vendor_product_by_source.csv'
'/opt/syslog-ng/etc/local_config/destinations/README.md' -> '/opt/syslog-ng/etc/conf.d/local/config/destinations/README.md'
'/opt/syslog-ng/etc/local_config/filters/README.md' -> '/opt/syslog-ng/etc/conf.d/local/config/filters/README.md'
'/opt/syslog-ng/etc/local_config/filters/example.conf' -> '/opt/syslog-ng/etc/conf.d/local/config/filters/example.conf'
'/opt/syslog-ng/etc/local_config/log_paths/README.md' -> '/opt/syslog-ng/etc/conf.d/local/config/log_paths/README.md'
'/opt/syslog-ng/etc/local_config/log_paths/lp-example.conf.tmpl' -> '/opt/syslog-ng/etc/conf.d/local/config/log_paths/lp-example.conf.tmpl'
'/opt/syslog-ng/etc/local_config/log_paths/lp-example.conf' -> '/opt/syslog-ng/etc/conf.d/local/config/log_paths/lp-example.conf'
'/opt/syslog-ng/etc/local_config/sources/README.md' -> '/opt/syslog-ng/etc/conf.d/local/config/sources/README.md'
syslog-ng checking config
sc4s version=v1.12.0
syslog-ng starting
Aug 16 11:44:12 hostname syslog-ng[1]: syslog-ng starting up; version='3.25.1'
Aug 16 11:44:12 hostname syslog-ng-config: sc4s version=v1.12.0
Aug 16 11:44:12 hostname syslog-ng[1]: curl: error sending HTTP request; url='https://http-singh-sudhir.splunkcloud.com:443/services/collector/event', error='Couldn\'t connect to server', worker_index='1', driver='d_hec_internal#0', location='/opt/syslog-ng/etc/conf.d/destinations/splunk_hec_internal.conf:2:5'
Aug 16 11:44:12 hostname syslog-ng[1]: Server disconnected while preparing messages for sending, trying again; driver='d_hec_internal#0', location='/opt/syslog-ng/etc/conf.d/destinations/splunk_hec_internal.conf:2:5', worker_index='1', time_reopen='10', batch_size='1'
Aug 16 11:44:12 hostname syslog-ng[1]: curl: error sending HTTP request; url='https://http-singh-sudhir.splunkcloud.com:443/services/collector/event', error='Couldn\'t connect to server', worker_index='0', driver='d_hec_internal#0', location='/opt/syslog-ng/etc/conf.d/destinations/splunk_hec_internal.conf:2:5'
Aug 16 11:44:12 hostname syslog-ng[1]: Server disconnected while preparing messages for sending, trying again; driver='d_hec_internal#0', location='/opt/syslog-ng/etc/conf.d/destinations/splunk_hec_internal.conf:2:5', worker_index='0', time_reopen='10', batch_size='1'

I am not able to understand what is missing here from my side. if is curl fails then it should be in both cases, looking forward to your help. please point out what is wrong with this.

Re: Failed to send log with syslog - syslog-ng[1]: curl: error sending HTTP request - 'Couldn\'t connect to server'

harsmarvania57 — Mon, 16 Aug 2021 14:02:12 GMT

Hi,

Looks like you are using very old version of SC4S, please use latest version.

Re: Failed to send log with syslog - syslog-ng[1]: curl: error sending HTTP request - 'Couldn\'t connect to server'

sutom — Mon, 16 Aug 2021 17:04:54 GMT

Thanks for the hit @harsmarvania57

As per suggestion I have updated the version and also manage the config according to new version, but again I am getting same kind of error.

update env_file.

[root@hostname ~]# cat /opt/sc4s/env_file SC4S_DEST_SPLUNK_HEC_DEFAULT_URL=https://http-singh-sudhir.splunkcloud.com:443 SC4S_DEST_SPLUNK_HEC_DEFAULT_TOKEN=Z93TSS87-F826-19V1-01W1-Q9Q8G1G8264 #Uncomment the following line if using untrusted SSL certificates #SC4S_DEST_SPLUNK_HEC_DEFAULT_TLS_VERIFY=no SC4S_DEST_GLOBAL_ALTERNATES=d_hec_debug

podman logs SC4S

[root@hostname sc4s]# podman logs SC4S curl: (7) Failed to connect to http-singh-sudhir.splunkcloud.com port 443: Connection timed out SC4S_ENV_CHECK_HEC: Invalid Splunk HEC URL, invalid token, or other HEC connectivity issue index=main. sourcetype=sc4s:fallback Startup will continue to prevent data loss if this is a transient failure. syslog-ng checking config sc4s version=1.86.4 starting goss starting syslog-ng Aug 16 16:07:35.327 hostname syslog-ng[166]: syslog-ng starting up; version='3.32.1' Aug 16 16:07:36.700 hostname syslog-ng[166]: curl: error sending HTTP request; url='http-singh-sudhir.splunkcloud.com:443/services/collector/event', error='Couldn\'t connect to server', worker_index='2', driver='d_hec_fmt#0', location='root generator dest_hec:5:5' Aug 16 16:07:36.700 hostname syslog-ng[166]: curl: error sending HTTP request; url='http-singh-sudhir.splunkcloud.com:443/services/collector/event', error='Couldn\'t connect to server', worker_index='3', driver='d_hec_fmt#0', location='root generator dest_hec:5:5' Aug 16 16:07:36.700 hostname syslog-ng[166]: Server disconnected while preparing messages for sending, trying again; driver='d_hec_fmt#0', location='root generator dest_hec:5:5', worker_index='2', time_reopen='10', batch_size='198' Aug 16 16:07:36.700 hostname syslog-ng[166]: Server disconnected while preparing messages for sending, trying again; driver='d_hec_fmt#0', location='root generator dest_hec:5:5', worker_index='3', time_reopen='10', batch_size='198'

Log from Debug file -

[root@hostname sc4s_events]# cat 2021-08-16-hec.log curl -k -u "sc4s HEC debug:$SC4S_DEST_SPLUNK_HEC_DEFAULT_TOKEN" "https://http-singh-sudhir.splunkcloud.com:443/services/collector/event" -d '{"time":"1629130055.327","sourcetype":"sc4s:events","source":"sc4s","index":"main","host":"sudhir4321","fields":{"sc4s_vendor_product":"sc4s_events","sc4s_syslog_facility":"syslog","sc4s_loghost":"sudhir4321","sc4s_container":"sudhir4321"},"event":"2021-08-16T16:07:35.327+00:00 sudhir4321 syslog-ng 166 - [meta sequenceId=\"1\"] syslog-ng starting up; version='3.32.1'"}' curl -k -u "sc4s HEC debug:$SC4S_DEST_SPLUNK_HEC_DEFAULT_TOKEN" "https://http-singh-sudhir.splunkcloud.com:443/services/collector/event" -d '{"time":"1629130056.401","sourcetype":"sc4s:events:startup:out","source":"sc4s","index":"main","host":"sudhir4321","fields":{"sc4s_vendor_product":"sc4s_events","sc4s_syslog_facility":"user","sc4s_loghost":"sudhir4321","sc4s_container":"sudhir4321"},"event":"syslog-ng-config: sc4s version=1.86.4"}' curl -k -u "sc4s HEC debug:$SC4S_DEST_SPLUNK_HEC_DEFAULT_TOKEN" "https://http-singh-sudhir.splunkcloud.com:443/services/collector/event" -d '{"time":"1629130056.700","sourcetype":"sc4s:events","source":"sc4s","index":"main","host":"sudhir4321","fields":{"sc4s_vendor_product":"sc4s_events","sc4s_syslog_facility":"syslog","sc4s_loghost":"sudhir4321","sc4s_container":"sudhir4321"},"event":"2021-08-16T16:07:36.700+00:00 sudhir4321 syslog-ng 166 - [meta sequenceId=\"3\"] curl: error sending HTTP request; url='https://http-singh-sudhir.splunkcloud.com:443/services/collector/event', error='Couldn\\'t connect to server', worker_index='2', driver='d_hec_fmt#0', location='root generator dest_hec:5:5'"}'

What could be the issue now, can you please help me to understand.

Re: Failed to send log with syslog - syslog-ng[1]: curl: error sending HTTP request - 'Couldn\'t connect to server'

s2_splunk — Mon, 16 Aug 2021 17:35:21 GMT

I don't think your HEC URL is correct if you are targeting a SplunkCloud stack.

It should be

https://http-inputs-<stackname>.splunkcloud.com

Re: Failed to send log with syslog - syslog-ng[1]: curl: error sending HTTP request - 'Couldn\'t connect to server'

sutom — Mon, 16 Aug 2021 17:39:48 GMT

Thanks for response @s2_splunk

I have sensitized the URL- my actual URL is-

"https://http-inputs-sudhir.splunkcloud.com:443"

Re: Failed to send log with syslog - syslog-ng[1]: curl: error sending HTTP request - 'Couldn\'t connect to server'

s2_splunk — Mon, 16 Aug 2021 17:45:10 GMT

Are you saying you modified all the log entries you have posted above to obfuscate the actual HEC endpoint?

If that's the case, I would try to test your Cloud Stacks's HEC functionality with a curl command from that machine:

curl -k https://http-inputs-<stack>.splunkcloud.com:443/services/collector/event -H "Authorization: Splunk <valid_token>" -d '{"event": "hello world"}'

And see if that works.

If it times out as well, something on your network doesn't allow the outbound connection.

Re: Failed to send log with syslog - syslog-ng[1]: curl: error sending HTTP request - 'Couldn\'t connect to server'

sutom — Tue, 17 Aug 2021 09:09:58 GMT

@s2_splunkyou are right. I have modified the HEC URL.

Here is the curl command response -

[root@hostname ~]# curl -k https://http-inputs-sudhir.splunkcloud.com:443/services/collector/event -H "Authorization: Splunk Z93TSS87-F826-19V1-01W1-Q9Q8G1G8264" -d '{"event": "hello world"}' {"text":"Success","code":0} [root@hostname ~]#

Output is success but with script is failing

Re: Failed to send log with syslog - syslog-ng[1]: curl: error sending HTTP request - 'Couldn\'t connect to server'

s2_splunk — Mon, 16 Aug 2021 22:47:18 GMT

Sorry, didn't see that you had already tried that.

Do you have any proxy configuration on your server that your podman container may not "see"?

Re: Failed to send log with syslog - syslog-ng[1]: curl: error sending HTTP request - 'Couldn\'t connect to server'

sutom — Tue, 17 Aug 2021 09:15:13 GMT

@s2_splunkI have tried the curl command and the output is given above.

Yes, we have a proxy configuration on the server. could you please help me with some hits - like how I can identify which proxy configuration affecting podman container.

Re: Failed to send log with syslog - syslog-ng[1]: curl: error sending HTTP request - 'Couldn\'t connect to server'

s2_splunk — Tue, 17 Aug 2021 15:59:36 GMT

Which version of podman are you running?

Re: Failed to send log with syslog - syslog-ng[1]: curl: error sending HTTP request - 'Couldn\'t connect to server'

sutom — Tue, 17 Aug 2021 16:38:19 GMT

podman -v
podman version 3.0.2-dev

Re: Failed to send log with syslog - syslog-ng[1]: curl: error sending HTTP request - 'Couldn\'t connect to server'

s2_splunk — Tue, 17 Aug 2021 18:37:49 GMT

Hmmm, not sure where to go from here. By default, podman should pick up system-wide proxy settings if this is really your issue.

On other thing you could try is to run the image in interactive mode and issue the curl command from within the container to see if that really is your issue.

You could also try explicitly setting your proxy variables in the env_file and see if that changes anything.

If you have a support entitlement, you can open a support case. Maybe others more familiar with podman and/or how to bypass your proxy for the node SC4S runs on have better ideas.

Re: Failed to send log with syslog - syslog-ng[1]: curl: error sending HTTP request - 'Couldn\'t connect to server'

sutom — Thu, 26 Aug 2021 19:10:06 GMT

Thanks @s2_splunk

As you suggested, I tried to run the CURL command in normal mode as well as image interactive mode and it got failed in interactive mode.

Normal mode CURL command -

[root@hostname ~]# curl -k https://http-inputs-sudhir.splunkcloud.com:443/services/collector -H "Authorization: Splunk Z93TSS87-F826-19V1-01W1-Q9Q8G1G8264" -d '{"event": "hello_world", "sourcetype":"mysourcetype"}' {"text":"Success","code":0}

CURL command in image interactive mode - it's timed out

podman exec -it containerid /bin/sh sh-4.4# curl -k -v https://http-inputs-sudhir.splunkcloud.com:443/services/collector -H "Authorization: Splunk Z93TSS87-F826-19V1-01W1-Q9Q8G1G8264" -d '{"event": "hello_world", "sourcetype":"mycontainer"}' * Trying 4.54.253.184... * TCP_NODELAY set * connect to 4.54.253.184 port 443 failed: Connection timed out * Trying 4.54.142.7... * TCP_NODELAY set ^C sh-4.4# exit

Looks like podman is not picking up system-wide proxy settings, Any suggestion here how I can troubleshoot or force podman for system proxy

Re: Failed to send log with syslog - syslog-ng[1]: curl: error sending HTTP request - 'Couldn\'t connect to server'

s2_splunk — Thu, 26 Aug 2021 22:51:16 GMT

Can you do an nslookup on your splunkcloud.com hostname and confirm that the IP address is the same as the one reported in your connection timeout?

I haven't run across this before. Typically, podman (and docker) should pick up system-wide proxy settings without a problem.

You can try to add the proxy variables to the env_file of your SC4S container. Get the current HTTPS_PROXY environment variable setting from your host and copy/paste into env_file. Restart/Retry.

Re: Failed to send log with syslog - syslog-ng[1]: curl: error sending HTTP request - 'Couldn\'t connect to server'

sutom — Fri, 27 Aug 2021 19:00:29 GMT

Thanks @s2_splunk

I did nslookup for my splunkcloud.com hostname and got the same IP address as in my connection timeout?

$ nslookup http-inputs-sudhir.splunkcloud.com Non-authoritative answer: Server: dns.google Address: 8.8.8.8 Name: sudhir-indexers-15287165932.xx-xxxxx-xx.elb.amazonaws.com Addresses: 4.54.253.184 4.54.142.7 Aliases: http-inputs-sudhir.splunkcloud.com

Then I set the HTTPS_PROXY = value(HTTPS_PROXY environment variable setting from my host) in env_file of your SC4S container and tried the CURL command in image interactive mode and it got

succeed - Thank you.

But again I got another issue related to CA certificate-

syslog-ng[165]: curl: error sending HTTP request; url='https://http-inputs-sudhir.splunkcloud.com:443/services/collector/event', error='Peer certificate cannot be authenticated with given CA certificates', worker_index='3', driver='d_hec_fmt#0', location='root generator dest_hec:5:5'

I have few SSL certificates installed on machine, but note sure whether those are creating problem.

Need suggestion, do I need to uninstall my ssl certs from here or need to install ssl root CA on cloud HEC-end point. I am hoping now this will not trouble more as earlier.

Re: Failed to send log with syslog - syslog-ng[1]: curl: error sending HTTP request - 'Couldn\'t connect to server'

sutom — Tue, 31 Aug 2021 07:00:06 GMT

Dear @s2_splunk ,

Finally I got my SC4S logs on the place,

Above error got resolve by setting the below variable-

SC4S_DEST_SPLUNK_HEC_TLS_VERIFY = no