topic Re: collectd reports "write_http plugin: curl_easy_perform failed with status 56: Recv failure: Connection reset by in Getting Data In

collectd reports "write_http plugin: curl_easy_perform failed with status 56: Recv failure: Connection reset by peer

richgalloway — Thu, 18 Jun 2020 21:40:46 GMT

I realize this may be more of a Linux problem than a Splunk problem, but I'm using code specifically for Splunk so perhaps someone here can help.

I compiled and installed collectd using the instructions at https://docs.splunk.com/Documentation/InfraApp/latest/Admin/ManageAgents. I have an HEC configured on my Splunk instance and can write to it.

curl -k https://1.2.3.4:8088/services/collector/raw -d "Testing" {"text":"Token is required","code":2}

Yes, I know I need a token in the curl command, but this at least demonstrates connectivity.

I've configured the write_http and write_splunk plugins correctly, I believe.

<Plugin write_http> <Node "example"> URL "http://1.2.3.4:8088/services/collector/raw" VerifyPeer false VerifyHost false Header "Header: Authorization: Splunk <redacted>" Format "JSON" Metrics true StoreRates true </Node> </Plugin> <Plugin write_splunk> server "1.2.3.4" port "8088" token "<redacted>" ssl true verifyssl false </Plugin>

As soon as collect starts it logs "write_http plugin: curl_easy_perform failed with status 56: Recv failure: Connection reset by peer" and does so repeatedly. No metrics are indexed by Splunk.

How do I fix this?

Re: collectd reports "write_http plugin: curl_easy_perform failed with status 56: Recv failure: Connection reset by

The_Simko — Fri, 19 Jun 2020 00:40:13 GMT

Howdy Rich,

Last time I did SAI, I replaced write_http with the items in write_splunk. So my conf didn't have any reference to write_http.

My conf has no record of write_http. Save off a copy of your config and try one without write_http?

From my notes:

FlushInterval 30

</LoadPlugin>

server “10.1.2.3”

port "8088"

token "<redacted>"

ssl true

verifyssl false

Dimension "entity_type:nix_host"

Dimension "role:webserver"

Dimension "InstanceId:i-whatevs"

</Plugin>

Re: collectd reports "write_http plugin: curl_easy_perform failed with status 56: Recv failure: Connection reset by

richgalloway — Fri, 19 Jun 2020 13:14:59 GMT

Thanks, Michael. Commenting-out write_http made the error message go away, but I'm still not getting any metric data.

Re: collectd reports "write_http plugin: curl_easy_perform failed with status 56: Recv failure: Connection reset by

richgalloway — Fri, 19 Jun 2020 17:18:27 GMT

I enabled debug logging and see the following. I looks like collectd is *trying* to write to splunk, but there appears to be nothing that suggests a failure to send.

[2020-06-19 17:06:57] [debug] plugin_dispatch_values: time = 1592586417.042; interval = 60.000; host = example; plugin = memory; plugin_instance = ; type = percent; type_instance = slab_recl; [2020-06-19 17:06:57] [debug] uc_insert: Added example/memory/memory-slab_recl to the cache. [2020-06-19 17:06:57] [debug] plugin: plugin_write: Writing values via write_splunk. [2020-06-19 17:06:57] [debug] plugin: plugin_write: Writing values via network. [2020-06-19 17:06:57] [debug] uc_insert: Added example/memory/percent-slab_recl to the cache. [2020-06-19 17:06:57] [debug] plugin: plugin_write: Writing values via write_splunk. [2020-06-19 17:06:57] [debug] plugin: plugin_write: Writing values via network. [2020-06-19 17:07:07] [debug] plugin_read_thread: Handling `flush/write_splunk'. [2020-06-19 17:07:07] [debug] plugin_read_thread: read-function of the `flush/write_splunk' plugin took 0.000004 seconds. [2020-06-19 17:07:07] [debug] plugin_read_thread: Effective interval of the `flush/write_splunk' plugin is 10.000 seconds. [2020-06-19 17:07:07] [debug] plugin_read_thread: Next read of the `flush/write_splunk' plugin at 1592586437.039.

Found this in splunkd.log, which appears to be significant. Not sure what to do about it since write_splunk should be sending the events in JSON format. It could, of course, be unrelated to my problem.

06-19-2020 17:15:17.054 +0000 ERROR MetricsProcessor - Failed to parse metrics input, most likely due to incorrect protocol JSON exception caught while processing collectd event: Unexpected character while looking for value: 'm' 06-19-2020 17:15:17.054 +0000 ERROR MetricsProcessor - JSON exception caught while processing collectd event: Unexpected character while looking for value: 'm'

Re: collectd reports "write_http plugin: curl_easy_perform failed with status 56: Recv failure: Connection reset by

sh1pit76 — Thu, 02 Jul 2020 17:41:10 GMT

I second this question.

I'm getting the same error:

RROR MetricsProcessor - JSON exception caught while processing collectd event: Unexpected character while looking for value: 'm'

The only way I've been able to make this disappear is to use the write_http plugin, but this breaks the SAI dashboard stats. Anyone have a solution to this?