So, what I am reading is that there is no way to avoid the double license hit if I ingest in two places, even if one is a subset of the other? Looking at outputs.conf, would it not be possible to configure the indexer to use the forwardedindex..blacklist = settings? ... View more

Thanks for the response! Much appreciated. I am trying to avoid allowing users to execute searches that will effectively go across the WAN to indexers that are not geographically close, mostly because I don't want or need super large result sets going from the indexer (in Asia, for example), all the way SH (in Europe, for example). I'd like to keep the central data center searches hitting central data center indexers, only, and the remote search heads hitting the remote indexers, only. Make sense? ... View more

Thanks @jshih_splunk . What I learned at .conf this year was a WHOLE lot about the itsi_refresh_queue. and itsi_consumer processes. This is really the root of the issue were were seeing. When defining and saving services with the entity filter, we were not seeing the association in the entity list immediately. In addition, we would get some odd results for KPIs, in that they included all entities for the whole service. As I now understand it, the refresh_queue and what's in there drives these associations. In our environment, a large delete or service creation would fill this queue up substantially and take a very long time to get worked off.... resulting in the associations not showing up in a timely manner. To fix this, we started added itsi_consumer processes in the inputs page. We are on a pretty beefy server and not currently in a SHC for ITSI. The additional consumer processes (75 believe it or not) we spun up resulted in that queue getting worked off faster. If that's not advised or if there are considerations with that type of setup that we should be aware of, let me know! ... View more

@esnyder. Thanks for the response. We've since upgraded to 4.1.2 and look forward to 4.4, but the issue still appears to remain. Specifically, when you set up recurring service imports (as upserts), the KPI and Service Id's appear to get regenerated each time the service import is completed. To test this, we had created a glass table based off a variety of KPIs for a service we knew would get 'upserted' in the next run. The service itself did not get duplicated (implying an upsert). However, the glass table reverted all KPIs to ad-hoc searches, the IDs for the service KPIs in the itsi_summary index were different, and the kpi history (in SA sparklines) gets zero'd out back to the most recent run. Could this simply be a bug? ... View more

Thanks @yannK. This makes good sense. In our case, I think the actual issue has more to do with your last bullet point. Specifically, the resulting base search is quite long with all of the host=a OR host=b OR host=c. If the parsing of that (which I'll check) is causing the bottle neck, that's probably part of the answer. As far as cardinality goes though, how would that affect performance or capacity planning? Our math on a per service basis looks something like 3 metrics x 3 kpis x 100 services x 40 entities = 36,000 for a given base search. Besides parsing, where else could a bottleneck exist for a base search like this? Note, the actual search (without all the extra stuff at the end) runs well under 30 seconds and only runs on a 15 minute frequency. ... View more

@skoelpin Thanks for the response. I didn't mean individual thresholds for each entity, I meant for the service aggregate, is there a different set of thresholds that get generated per service. We have services using templates (both service and threshold templates). It seems that the threshold template is generating some thresholds based on something (not sure what) and applying that to all services. I would have thought that each service for which the template was applied would get a different adaptive threshold set applied to it. What am I missing? Thanks! ... View more

@skoelpin Thanks for the response. I didn't mean we were looking for adaptive thresholding on the entities. What I meant was that the adapting thresholding that is turned on in the threshold template does not appear to be creating unique thresholds adaptations for each of the services (which use service templates). It seems like it is creating a set of thresholds once a day for some large data set then applying that to all services that have the threshold template attached. That doesn't seem right to me. What am I missing? Thanks! ... View more

Have there been any updates to this issue? We too have services synced to Service Templates and thresholding linked to a Threshold template that employs adaptive thresholding. I would have thought that the adaptive thresholding would be applied to each service by learning the trend on the entities filtered in to that service (and not have to manually set each service threshold) but it does not appear to be working that way. When using a Service Template and Threshold Templates with adaptive thresholding, what is used for training and does it uniquely apply to each service employing those templates? ... View more

@VatsalJagani Thanks for that. I have read that about 15 times. It all makes sense. I guess it means that the lag you are accounting for is done via the event timestamp not the index time. In 3.1.4, this is limited to 1799 seconds in the UI. My question is what is the best approach to account for lags that are longer than this. In a base search do I just use earliest and latest? Do I hack the shared saved search that is resultant from the base search? ... View more

To clarify.... On the entities page, there is an Associated Service column. Most of my entities have an association to a service. Does anyone know how or when this occurs? I did not specify the association manually (other than via entity filters in the service definition). Thanks ... View more

This might work. Does anyone know what ITSI uses in its lag calculations? Index time or timestamp? For example, if I specify a search lag of 10 minutes, is that searching for timestamps 10 minutes ago or index times 10 minutes ago? Will make a big difference. ... View more

I get an error that says "Dispatch COmmand: The number of search artifacts in the dispatch directory is higher than recommended...." I am just trying to figure out the best way to determine what is driving the large number of artifacts specifically. (Since we are running ITSI, I'm wondering what part of that, if any is contributing to the issue). ... View more

Sorry, it's on a different system. Is it possible that there are performance limits in rendering the sparklines for some reason? What I am also learning, I think, is that the sparklines reflect values that correspond to info_min_time and info_max_time in the itsi_summary index, NOT the time that the KPI was run. So, if I have a KPI that looks at data from 45-60 minutes ago, the data point appears at a location on that sparkline that seems to match one of those two times, not the KPI run time. I imagine this contributes to the problem a little as well. ... View more