collectd reports "write_http plugin: curl_easy_perform failed with status 56: Recv failure: Connection reset by peer

[richgalloway]

I realize this may be more of a Linux problem than a Splunk problem, but I'm using code specifically for Splunk so perhaps someone here can help.

I compiled and installed collectd using the instructions at https://docs.splunk.com/Documentation/InfraApp/latest/Admin/ManageAgents.  I have an HEC configured on my Splunk instance and can write to it.

curl -k https://1.2.3.4:8088/services/collector/raw -d "Testing"
{"text":"Token is required","code":2}

Yes, I know I need a token in the curl command, but this at least demonstrates connectivity.

I've configured the write_http and write_splunk plugins correctly, I believe.

<Plugin write_http>
        <Node "example">
                URL "http://1.2.3.4:8088/services/collector/raw"
                VerifyPeer false
                VerifyHost false
                Header "Header: Authorization: Splunk <redacted>"
                Format "JSON"
                Metrics true
                StoreRates true
        </Node>
</Plugin>

<Plugin write_splunk>
        server "1.2.3.4"
        port "8088"
        token "<redacted>"
        ssl true
        verifyssl false
</Plugin>

As soon as collect starts it logs "write_http plugin: curl_easy_perform failed with status 56: Recv failure: Connection reset by peer" and does so repeatedly.  No metrics are indexed by Splunk.

How do I fix this?

[The_Simko]

Howdy Rich,

Last time I did SAI, I replaced write_http with the items in write_splunk. So my conf didn't have any reference to write_http.  
  
My conf has no record of write_http.  Save off a copy of your config and try one without write_http?


From my notes:

<LoadPlugin "write_splunk">

        FlushInterval 30

</LoadPlugin>

<Plugin write_splunk>

           server “10.1.2.3”

           port "8088"

           token "<redacted>"

           ssl true

           verifyssl false

           Dimension "entity_type:nix_host"

           Dimension "role:webserver"

           Dimension "InstanceId:i-whatevs"

</Plugin>

[richgalloway]

Thanks, Michael. Commenting-out write_http made the error message go away, but I'm still not getting any metric data.

[richgalloway]

I enabled debug logging and see the following.  I looks like collectd is *trying* to write to splunk, but there appears to be nothing that suggests a failure to send.

[2020-06-19 17:06:57] [debug] plugin_dispatch_values: time = 1592586417.042; interval = 60.000; host = example; plugin = memory; plugin_instance = ; type = percent; type_instance = slab_recl;
[2020-06-19 17:06:57] [debug] uc_insert: Added example/memory/memory-slab_recl to the cache.
[2020-06-19 17:06:57] [debug] plugin: plugin_write: Writing values via write_splunk.
[2020-06-19 17:06:57] [debug] plugin: plugin_write: Writing values via network.
[2020-06-19 17:06:57] [debug] uc_insert: Added example/memory/percent-slab_recl to the cache.
[2020-06-19 17:06:57] [debug] plugin: plugin_write: Writing values via write_splunk.
[2020-06-19 17:06:57] [debug] plugin: plugin_write: Writing values via network.
[2020-06-19 17:07:07] [debug] plugin_read_thread: Handling `flush/write_splunk'.
[2020-06-19 17:07:07] [debug] plugin_read_thread: read-function of the `flush/write_splunk' plugin took 0.000004 seconds.
[2020-06-19 17:07:07] [debug] plugin_read_thread: Effective interval of the `flush/write_splunk' plugin is 10.000 seconds.
[2020-06-19 17:07:07] [debug] plugin_read_thread: Next read of the `flush/write_splunk' plugin at 1592586437.039.

Found this in splunkd.log, which appears to be significant.  Not sure what to do about it since write_splunk should be sending the events in JSON format. It could, of course, be unrelated to my problem.

06-19-2020 17:15:17.054 +0000 ERROR MetricsProcessor - Failed to parse metrics input, most likely due to incorrect protocol JSON exception caught while processing collectd event: Unexpected character while looking for value: 'm'
06-19-2020 17:15:17.054 +0000 ERROR MetricsProcessor - JSON exception caught while processing collectd event: Unexpected character while looking for value: 'm'

[sh1pit76]

I second this question.

I'm getting the same  error:

RROR MetricsProcessor - JSON exception caught while processing collectd event: Unexpected character while looking for value: 'm'

The only way I've been able to make this disappear is to use the write_http plugin, but this breaks the SAI dashboard stats. Anyone have a solution to this?