Thanks @jshih_splunk . What I learned at .conf this year was a WHOLE lot about the itsi_refresh_queue. and itsi_consumer processes. This is really the root of the issue were were seeing. When defining and saving services with the entity filter, we were not seeing the association in the entity list immediately. In addition, we would get some odd results for KPIs, in that they included all entities for the whole service. As I now understand it, the refresh_queue and what's in there drives these associations. In our environment, a large delete or service creation would fill this queue up substantially and take a very long time to get worked off.... resulting in the associations not showing up in a timely manner.
To fix this, we started added itsi_consumer processes in the inputs page. We are on a pretty beefy server and not currently in a SHC for ITSI. The additional consumer processes (75 believe it or not) we spun up resulted in that queue getting worked off faster.
If that's not advised or if there are considerations with that type of setup that we should be aware of, let me know!
... View more