Splunk App for Infrastructure, CentOS 7, JSON exception caught while processing collectd event: Unexpected character: 'm'

[asucrews]

Both servers is CentOS 7
One with Splunk Enterprise 7.2.5
Splunk App for Infrastructure 1.2.3
Splunk Add-on for Infrastructure 1.2.3

one with Splunk Universal Forwarder 7.2.5

Error message: 03-20-2019 13:52:05.257 -0700 ERROR MetricsProcessor - JSON exception caught while processing collectd event: Unexpected character: 'm'
03-20-2019 13:52:05.257 -0700 ERROR MetricsProcessor - Failed to parse metrics input, most likely due to incorrect protocol JSON exception caught while processing collectd event: Unexpected character: 'm'
"

Hello, I am trying out Splunk App for Infrastructure and at this time all i am getting is the above error message. And i not sure if something changed in collectd or app. I have read thought the docs online but can't seem to find any thing like this. Has anyone ran in to this issue before.

[asucrews]

I have narrowed it down the issue to write_splunk plugin, when i switch to write_http plugin i start getting data. Not sure if this effect the dashboards but it is now working.

[dagarwal_splunk]

I have seen the same issue before. The issue was with the hec_token not being set properly with correct sourcetype OR Add on for Infra not installed. Are you sending collectd data directly to SAI or using any forwarder in between?

For write_http to work for you, it needs collectd_http as the sourcetype. Did you change that after switching to write_http?

[dagarwal_splunk]

Sourcetype should be em_metrics not Automatic.

[asucrews]

disabled ssl, and change sourcetype to em_metrics. still getting "03-21-2019 09:13:51.651 -0700 ERROR MetricsProcessor - JSON exception caught while processing collectd event: Unexpected character: 'm'"
"03-21-2019 09:13:51.651 -0700 ERROR MetricsProcessor - Failed to parse metrics input, most likely due to incorrect protocol JSON exception caught while processing collectd event: Unexpected character: 'm'"

[dagarwal_splunk]

Could you please check the hec token that you created?

Make sure sourcetype and index is "em_metrics" for the token.

https://docs.splunk.com/Documentation/InfraApp/1.2.3/Install/Install

[asucrews]

Deleted are readded HED per https://docs.splunk.com/Documentation/InfraApp/1.2.3/Install/Install still same error

[asucrews]

collectd 5.8.1, http://collectd.org/
by Florian octo Forster
for contributions see `AUTHORS'

[dagarwal_splunk]

Did you run the script on your terminal from "Add Data" page on Splunk App for Infra ?

[dagarwal_splunk]

Did you restart Splunk after installing "Splunk Add on for Infra" ?

[dagarwal_splunk]

ALso, rerun the script with right hec_token if you have deleted and created a new token

[asucrews]

deleted and recreated HEC . still same error.

Review
Input Type Token
Name SPI_HEC
Source name override N/A
Description N/A
Enable indexer acknowledgements No
Output Group N/A
Allowed indexes em_metrics
Default index em_metrics
Source Type Automatic
App Context splunk_app_infrastructure