Looks like you don't have "Add-on for Infrastructure" installed for Splunk App for Infra. Could you please confirm? Follow documentation for both Windows and Linux monitoring: https://docs.splunk.com/Documentation/InfraApp/1.2.2/Install/Install ... View more

Can you try some things to debug this ? On your Windows Machine, check if UF is actually sending data to SAI (Splunk App for Infrastructure): ${SPLUNK_HOME}/bin/splunk list forward-server If yes, On your SAI Instance, run this search (check if SAI has the metrics data in the em_metrics index): | mcatalog values("host") , values("_dims") as "dims" WHERE metric_name=processor.* AND index=em_metrics BY "host" | table host ... View more

What I found is that when I stopped the process that I was monitoring , all the values like name.ps_code/ps_data etc becomes 0 from positive value. Maybe set an alert on any of this value to monitor it? ... View more

I tried it and understand your question now. ... View more

Sorry, I didn't completely get your follow-up question. I guess once you configure collectd.conf and add all the processes you want to monitor, you should be able to see it in SAI Analysis page and you can monitor like setting up alerts? ... View more

@qhmassc For Linux/Unix, you can use collectd's "Processes" plugin and send processes metrics data to SAI. Just update your collectd.conf file and add this plugin and restart collectd. You can find lot of info on how to add this plugin with a quick search. ... View more

Currently we don't have any alerting feature in SAI/SII based on activity of the server. We will be adding this feature in the future release. ... View more

Great! you can add your dimensions back now if you want.. ... View more

Maybe try this fix: https://answers.splunk.com/answers/453888/after-configuring-the-http-event-collector-why-am.html ... View more

Let's check /opt/splunk/var/log/splunk/splunkd.log for any indexing error like " metric event not indexed". Also, can you try this curl from Linux host as well mentioned in doc: (https://docs.splunk.com/Documentation/Splunk/7.2.3/Data/Sendmetricstoametricsindex) curl -k https://localhost:8088/services/collector \ -H "Authorization: Splunk b0221cd8-c4b4-465a-9a3c-273e3a75aa29" \ -d '{"time": 1486683865.000,"source":"disk","host":"host_99","fields":{"region":"us-west-1","datacenter":"us-west-1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":1099511627776,"metric_name":"total"}}' Replace token with yours and localhost with your SAI instance. See, if you get success. ... View more

Formatting got messed up when you pasted collectd.conf file here. ... View more

It seems you put inside Plugin write_splunk - Dimension "entity_type:Linux_Host" ... View more

Your token should look like : "xxx2fb15-5c36-489e-b624-xyzgdshg" . Not the name of token? ... View more

Maybe post your collectd.conf file here hiding any personal info with xxxxx. ... View more

Let's try to debug: Check collectd again: 1. collectd service is running. 2. No errors in collectd.log file 3. Plugin write_splunk is there in collectd.conf file. 4. The server (SAI server), token, port are correct for write_splunk in collectd.conf file Check SAI again: 1. Add-on for Infra installed 2. Token used by collectd is enabled and use em_metrics for sourcetype and index. ... View more

might need to restart ... View more

It should work now. You can try the search again? ... View more

https://docs.splunk.com/Documentation/InfraApp/1.2.2/Install/Install See step 5. You can use the Splunk Web. What token are you using right now in collectd.conf file? ... View more

Also, what is the sourcetype and index for your hec_token? Check hec_token in your Splunk instance that is used by collectd. Make sure both are "em_metrics". ... View more

Try this search and see if it returns anything: | mstats avg(_value) WHERE index=em_metrics AND metric_name=cpu.* by host ... View more

What about the log file for collectd ? Also, What OS version you are using? ... View more

Are there any errors in /etc/collectd/collectd.log file ? Something like "curl failed" ? ... View more

Do you have all 3 required arguments (non empty) in the generated script that you run: SPLUNK_URL, HEC_PORT, HEC_TOKEN? If yes, can you paste the generated script here (hide(*) any private info in the script) ? ... View more

You should check the inputs.conf file on your UF. Make sure you have the perfmon stanzas like this : [perfmon://CPU Load] counters = % C1 Time;% C2 Time;% Idle Time;% Processor Time;% User Time;% Privileged Time;% Reserved Time;% Interrupt Time instances = * interval = 30 object = Processor index = em_metrics _meta = entity_type::Windows_Host You will need the "Processor" object for entity to be discovered. index=em_metrics and _meta = entity_type::Windows_Host are required. ... View more

Try some of these steps first : 1) Can you check if the UF on the Windows machine is actually sending data? ".\splunk list forward-server". If you don't have any user login created on UF, you can use this: https://docs.splunk.com/Documentation/Splunk/7.2.1/Admin/User-seedconf 2) Make sure "outputs.conf" in UF is sending data to Infrastructure App. 3) Use Enterprise search to check if Windows metrics data is coming. Something like: " | mstats avg(_value) WHERE metric_name=* AND index=em_metrics AND entity_type=Windows_Host by metric_name" ... View more