topic Re: Collectd throwing ssl errors from one server, but not others in All Apps and Add-ons

Collectd throwing ssl errors from one server, but not others

zthomas — Wed, 16 Oct 2019 15:43:49 GMT

I have installed and configured collectd on 3 servers so far, and on 2 of them it works fine. These servers show up in the Splunk App for Infrastructure correctly; I can view all metrics as expected. The third server is throwing two errors consistently, as shown below.

Oct 16 10:37:54 <hostname> collectd[13731]: plugin_read_thread: read-function of the `flush/write_splunk' plugin took 90.018 seconds, which is above its read interval (90.000 seconds). You might want to adjust the `Interval' or `ReadThreads' settings.
Oct 16 10:38:54 <hostname> collectd[13731]: write splunk plugin: curl_easy_perform failed to connect to <HEC VIP>:8088 with status 35: SSL connect error

The collectd.conf is literally copy-pasted between all three servers, and only the Hostname option has been updated.

The HEC VIP is a load balancer pointing to all 4 indexers in our indexer cluster, if that makes a difference.

Re: Collectd throwing ssl errors from one server, but not others

dagarwal_splunk — Wed, 30 Sep 2020 02:33:44 GMT

Can you provide Linux distro?
Also can you try this from the non working server? Check if you get "Success". https://docs.splunk.com/Documentation/Splunk/7.3.1/Metrics/GetMetricsInOther Use curl to get some fake data in. Update server, port & token:

curl -k https://localhost:8088/services/collector -H "Authorization: Splunk b0221cd8-c4b4-465a-9a3c-273e3a75aa29" -d '{"time": 1486683865.000,"event":"metric","source":"disk","host":"host_99","fields":{"region":"us-west-1","datacenter":"us-west-1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":1099511627776,"metric_name":"total"}}'

Re: Collectd throwing ssl errors from one server, but not others

zthomas — Wed, 16 Oct 2019 16:29:50 GMT

CentOS Linux release 7.4.1708 (Core)

I would need to replace the localhost with the HEC vip and the b0221cd8-c4b4-465a-9a3c-273e3a75aa291 with my HEC token, no?

Re: Collectd throwing ssl errors from one server, but not others

dagarwal_splunk — Wed, 16 Oct 2019 16:32:07 GMT

yes... update them..

Re: Collectd throwing ssl errors from one server, but not others

zthomas — Wed, 16 Oct 2019 16:38:57 GMT

I ran the curl command you posted (add -v) and got the below error:

* About to connect() to <HEC VIP> port 8088 (#0)
*   Trying <HEC VIP>...
* Connected to <HEC VIP> (<HEC VIP>) port 8088 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* NSS error -5938 (PR_END_OF_FILE_ERROR)
* Encountered end of file
* Closing connection 0
curl: (35) Encountered end of file

This took 1m0.013s to return.

Re: Collectd throwing ssl errors from one server, but not others

dagarwal_splunk — Wed, 16 Oct 2019 16:42:09 GMT

It looks you need to update curl or nss?
Update and try same thing again?

Re: Collectd throwing ssl errors from one server, but not others

dagarwal_splunk — Wed, 16 Oct 2019 16:55:04 GMT

This curl should give you "Success" from the servers that are working for you. Try it out..
I think you need to fix this nss error for collectd to work for you..

Re: Collectd throwing ssl errors from one server, but not others

zthomas — Wed, 16 Oct 2019 16:59:11 GMT

I'm working with my sys admin team to get the nss issue sorted, will report back once that is done and another curl is run.

Re: Collectd throwing ssl errors from one server, but not others

bashby_splunk — Mon, 21 Oct 2019 23:03:22 GMT

Are you sending data to HEC over HTTP with SSL enabled in collectd.conf? If you are, disable SSL in collectd.conf. Set ssl to false in the write_splunk plug-in.

Re: Collectd throwing ssl errors from one server, but not others

mpise_splunk — Wed, 27 Nov 2019 19:37:48 GMT

Check the box for Enable SSL in HEC Global settings.

Re: Collectd throwing ssl errors from one server, but not others

schwagem — Thu, 16 Jul 2020 18:45:24 GMT

I got here because I was getting errors from my one and only server, using collectd and Splunk App for Infrastructure.

The global SSL needs to be checked. If not, you will get the SSL error mentioned. Then, on the host running collectd, you need to ensule that the write_splunk stanza says "ssl true" in /etc/collectd.conf. If it says false, you'll see the following error:

write splunk plugin: curl_easy_perform failed to connect to your_indexer_host:8088 with status 56: Failure when receiving data from the peer

So, at this day and age (Splunk 7.3.3 with Universal Forwarder 8.0.1 on my client, and Splunk App for Infrastructure 2.1.0, ensure that the /etc/collectd.conf is correct, and that Globally, you have SSL checked (go to Settings-> Data, and hit the Global Settings box in the upper right hand corner)..