ITSI - Issue with no entities showing in service

[rcraft1218]

Hello,

I'm currently having an issue with a new Splunk ITSI installation with entities not showing up in the service after KPI's are added. I recently completed the Implementing Splunk ITSI class they offer and set everything up according to how we did it in that class. Here's what I've gone through so far:

1. Added all of my entities via the saved search that imports based on forwarders connected. This successfully identified and added all of my devices as entities (over 300 devices)
2. I cloned the CPU and Memory searches that are built-in, verified the searches returned results, and then set the lag time to what was recommended by the search (348 seconds in my case)
3. I then created the service, added my entities and linked my KPI Base Searches through the Generic KPI option.
4. When setting the threshold of the KPI the Splunk instance was able to pull data for the entities over the last 60 minutes for what the KPI Base Search was supposed to be pulling
5. Once I save and save and enable the service if I go to "View Health" it says no entities are available and all of the KPI's show N/A. Even though all of the searches work individually, and my entities are added and linked to the service (verified per the Entities page) there is nothing displayed here.

I've gone through all of the troubleshooting steps on the Splunk Wiki, verified that all of the Splunk for Linux/Unix options I need are enabled, sysstat is available on the servers, and so on. As far as I can see, everything looks correct. However, I cannot get past this issue and while I've reached out to Splunk Support on it they refuse to assist with even basic troubleshooting and want to charge me administrative fees. So, in my desperation I'm reaching out here to the community in the hope of getting some assistance with this issue.

Thanks in advance. If you need any other information please don't hesitate to let me know.

[yannK]

2 remarks : 348 seconds of indexing delay, this is not great. you want to be under a minute. Maybe your servers clocks are drifting ...

When you run the KPI searches do you get results, this will be a good test.

Are your KPIs linked to Shared bases searches (SBS)?
Do you use a split by entity, how many entities total would be indirectly linked to a single SBS ? (10000 is usually the cadinality limit by default)
Try to convert to a hadhoc KPI in a single service to compare.