Splunk IT Service Intelligence

Splunk App for Infrastructure, CentOS 7, JSON exception caught while processing collectd event: Unexpected character: 'm'

Path Finder

Both servers is CentOS 7
One with Splunk Enterprise 7.2.5
Splunk App for Infrastructure 1.2.3
Splunk Add-on for Infrastructure 1.2.3

one with Splunk Universal Forwarder 7.2.5

Error message: 03-20-2019 13:52:05.257 -0700 ERROR MetricsProcessor - JSON exception caught while processing collectd event: Unexpected character: 'm'
03-20-2019 13:52:05.257 -0700 ERROR MetricsProcessor - Failed to parse metrics input, most likely due to incorrect protocol JSON exception caught while processing collectd event: Unexpected character: 'm'
"

Hello, I am trying out Splunk App for Infrastructure and at this time all i am getting is the above error message. And i not sure if something changed in collectd or app. I have read thought the docs online but can't seem to find any thing like this. Has anyone ran in to this issue before.

0 Karma

Path Finder

I have narrowed it down the issue to writesplunk plugin, when i switch to writehttp plugin i start getting data. Not sure if this effect the dashboards but it is now working.

0 Karma

Splunk Employee
Splunk Employee

I have seen the same issue before. The issue was with the hec_token not being set properly with correct sourcetype OR Add on for Infra not installed. Are you sending collectd data directly to SAI or using any forwarder in between?

For writehttp to work for you, it needs collectdhttp as the sourcetype. Did you change that after switching to write_http?

0 Karma

Splunk Employee
Splunk Employee

Sourcetype should be em_metrics not Automatic.

0 Karma

Path Finder

disabled ssl, and change sourcetype to em_metrics. still getting "03-21-2019 09:13:51.651 -0700 ERROR MetricsProcessor - JSON exception caught while processing collectd event: Unexpected character: 'm'"
"03-21-2019 09:13:51.651 -0700 ERROR MetricsProcessor - Failed to parse metrics input, most likely due to incorrect protocol JSON exception caught while processing collectd event: Unexpected character: 'm'"

0 Karma

Splunk Employee
Splunk Employee

Could you please check the hec token that you created?

Make sure sourcetype and index is "em_metrics" for the token.

https://docs.splunk.com/Documentation/InfraApp/1.2.3/Install/Install

0 Karma

Path Finder

Deleted are readded HED per https://docs.splunk.com/Documentation/InfraApp/1.2.3/Install/Install still same error

0 Karma

Path Finder

collectd 5.8.1, http://collectd.org/
by Florian octo Forster
for contributions see `AUTHORS'

0 Karma

Splunk Employee
Splunk Employee

Did you run the script on your terminal from "Add Data" page on Splunk App for Infra ?

0 Karma

Splunk Employee
Splunk Employee

Did you restart Splunk after installing "Splunk Add on for Infra" ?

0 Karma

Splunk Employee
Splunk Employee

ALso, rerun the script with right hec_token if you have deleted and created a new token

0 Karma

Path Finder

deleted and recreated HEC . still same error.

Review
Input Type Token
Name SPIHEC
Source name override N/A
Description N/A
Enable indexer acknowledgements No
Output Group N/A
Allowed indexes em
metrics
Default index emmetrics
Source Type Automatic
App Context splunk
app_infrastructure

0 Karma