Getting Data In

collectd reports "write_http plugin: curl_easy_perform failed with status 56: Recv failure: Connection reset by peer

richgalloway
SplunkTrust
SplunkTrust

I realize this may be more of a Linux problem than a Splunk problem, but I'm using code specifically for Splunk so perhaps someone here can help.

I compiled and installed collectd using the instructions at https://docs.splunk.com/Documentation/InfraApp/latest/Admin/ManageAgents.  I have an HEC configured on my Splunk instance and can write to it.

 

curl -k https://1.2.3.4:8088/services/collector/raw -d "Testing"
{"text":"Token is required","code":2}

 

Yes, I know I need a token in the curl command, but this at least demonstrates connectivity.

I've configured the write_http and write_splunk plugins correctly, I believe.

 

<Plugin write_http>
        <Node "example">
                URL "http://1.2.3.4:8088/services/collector/raw"
                VerifyPeer false
                VerifyHost false
                Header "Header: Authorization: Splunk <redacted>"
                Format "JSON"
                Metrics true
                StoreRates true
        </Node>
</Plugin>

<Plugin write_splunk>
        server "1.2.3.4"
        port "8088"
        token "<redacted>"
        ssl true
        verifyssl false
</Plugin>

 

As soon as collect starts it logs "write_http plugin: curl_easy_perform failed with status 56: Recv failure: Connection reset by peer" and does so repeatedly.  No metrics are indexed by Splunk.

How do I fix this?

---
If this reply helps you, Karma would be appreciated.
Labels (1)
Tags (2)
0 Karma

The_Simko
Path Finder

Howdy Rich,

 

Last time I did SAI, I replaced write_http with the items in write_splunk. So my conf didn't have any reference to write_http.  
  
My conf has no record of write_http.  Save off a copy of your config and try one without write_http?


From my notes:

<LoadPlugin "write_splunk">

        FlushInterval 30

</LoadPlugin>



<Plugin write_splunk>

           server “10.1.2.3”

           port "8088"

           token "<redacted>"

           ssl true

           verifyssl false

           Dimension "entity_type:nix_host"

           Dimension "role:webserver"

           Dimension "InstanceId:i-whatevs"

</Plugin>

richgalloway
SplunkTrust
SplunkTrust
Thanks, Michael. Commenting-out write_http made the error message go away, but I'm still not getting any metric data.
---
If this reply helps you, Karma would be appreciated.
0 Karma

richgalloway
SplunkTrust
SplunkTrust

I enabled debug logging and see the following.  I looks like collectd is *trying* to write to splunk, but there appears to be nothing that suggests a failure to send.

[2020-06-19 17:06:57] [debug] plugin_dispatch_values: time = 1592586417.042; interval = 60.000; host = example; plugin = memory; plugin_instance = ; type = percent; type_instance = slab_recl;
[2020-06-19 17:06:57] [debug] uc_insert: Added example/memory/memory-slab_recl to the cache.
[2020-06-19 17:06:57] [debug] plugin: plugin_write: Writing values via write_splunk.
[2020-06-19 17:06:57] [debug] plugin: plugin_write: Writing values via network.
[2020-06-19 17:06:57] [debug] uc_insert: Added example/memory/percent-slab_recl to the cache.
[2020-06-19 17:06:57] [debug] plugin: plugin_write: Writing values via write_splunk.
[2020-06-19 17:06:57] [debug] plugin: plugin_write: Writing values via network.
[2020-06-19 17:07:07] [debug] plugin_read_thread: Handling `flush/write_splunk'.
[2020-06-19 17:07:07] [debug] plugin_read_thread: read-function of the `flush/write_splunk' plugin took 0.000004 seconds.
[2020-06-19 17:07:07] [debug] plugin_read_thread: Effective interval of the `flush/write_splunk' plugin is 10.000 seconds.
[2020-06-19 17:07:07] [debug] plugin_read_thread: Next read of the `flush/write_splunk' plugin at 1592586437.039.

Found this in splunkd.log, which appears to be significant.  Not sure what to do about it since write_splunk should be sending the events in JSON format. It could, of course, be unrelated to my problem.

06-19-2020 17:15:17.054 +0000 ERROR MetricsProcessor - Failed to parse metrics input, most likely due to incorrect protocol JSON exception caught while processing collectd event: Unexpected character while looking for value: 'm'
06-19-2020 17:15:17.054 +0000 ERROR MetricsProcessor - JSON exception caught while processing collectd event: Unexpected character while looking for value: 'm'

 

---
If this reply helps you, Karma would be appreciated.
0 Karma

sh1pit76
Explorer

I second this question.

I'm getting the same  error:

RROR MetricsProcessor - JSON exception caught while processing collectd event: Unexpected character while looking for value: 'm'

 

The only way I've been able to make this disappear is to use the write_http plugin, but this breaks the SAI dashboard stats. Anyone have a solution to this? 

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...