To conclude the steps for resolving Metrics data (collectd) collection issues:
(NOTE: machine2 is the monitored Linux machine i.e running collectd and Splunk UF; machine 1 is running SAI And SAI Add-on)
Check if collectd running or installed on monitored Machine 2..
apt-cache policy collectd
ps -ef | grep collectd
Check Metrics data coming in: | mcatalog values(metric_name) WHERE host=${Machine 2} AND index=em_metrics
Machine2: Do you see any recurring errors like "curl_easy_perform failed" in collectd.log or any other error?
In Machine 1, Check if all the Hec tokens are enabled: Settings -> Data Inputs ->HTTP Event Collector
Machine 1: Check the Global Settings on the same page as 2. Verify "enable ssl" is checked, "Use Deployment Server" unchecked and note down the port number.
Machine 1: Verify the HEC token you are using has default index as "em_metrics"
Now In Machine 2, check /etc/collectd/collectd.conf file. Verify that HEC token, server and port number in write_splunk stanza is correct.
Try sending fake data from Machine 2 to Machine 1 using curl and see if you get success. Here is the curl command you need to run in Machine 2:
curl -k https://Machine1:8088/services/collector -H "Authorization: Splunk hec_token_here" -d '{"time": 1486683865.000,"event":"metric","source":"disk","host":"host_99","fields":{"region":"us-west-1","datacenter":"us-west-1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":1099511627776,"metric_name":"total"}}'
You should see "Success" Message. If not, try to fix the error message that you get.
https://docs.splunk.com/Documentation/Splunk/7.3.3/Metrics/GetMetricsInOther#Example_of_sending_metrics_using_HEC
Update token, port and server in the command
... View more