So first thing to check is that the agg policies you have to group are not excluding events. Skipped events is relevant to grouped events only so when you see skipped events, are they events that are not being grouped by the agg policies.  You can test this by making sure your default policy is grouping by 'source' only.  Also can you tell me if you have tsix enabled?  There is a known issue with large number of NE's being created if you have that enabled.  Troubleshooting, turn off corr searches except for one, and one agg policy.  See if events get grouped and skipped searches reduce.  If so, then you need to analyze what your agg policy is doing that is excluding those events.  Basically it's the process of elimination and remembering skipped events means, skipped grouping, not skipped creating NE's. ... View more

No - retention is based on a per index basis only - the site an index is located at can not determine retention settings Given that, sadly, the second part of your question about search affinity is not possible as a result. ... View more

Legally, no, you have to pay for the amount of data ingested. If you ingest it twice (on the same indexer or different indexers), you pay for it twice. If you blacklist an index, then you won't forward that data and won't be charged for it twice. But I don't think that's what you're looking for (correct me if I'm misinterpreting your request). If it were me, I would see if having the central SHC reaching out to the remote indexers delivers reasonable performance (remember, the slowest indexer to respond will affect the time gof the search), and what data you really need forwarded. Beware of things that might consume your bandwidth unnecessarily (do you need _internal data forwarded? It's free, but can be big if you have a large environment). You also haven't mentioned what kind of use you have for the data - if you're running Enterprise Security, your performance for your data summarization might be terrible if run over a WAN (and you can't use the summaries generated by the remote site, as each set of accelerated data is accessible by only one search head cluster). You haven't really said much about quantity of data, and if you want all the data searchable at both sites. ... View more

If you have an ITSI instance, you probably have a support contract, please open a support case for deeper troubleshooting. From the description of the question, here are some pointers. the itsi_refresh_queue is a kvstore collection used to store the changes to apply to ITSI, it can fill up when you have service-templates change to propagate, entity import, shared-bases-searches update, threshold updates... If you had a recent mass change or an upgrade, this could be the reason. after a refresh task has being applied, an object will be removed from the collection. However the kvstore is a file storage, when a collection is growing, it will add new chunks of files on disk to reserve disk space. But when the objects are removed from the collection, the files on disk will not be removed ( instead, the reserved slots we be reused later). So the disk space will not be recovered. This may be what happened to you, a large collection grew big and did not shrink when the objects were removed. To check : You can check the collection size on the ITSI healthcheck dashboard, look at the "KV Store Collections" and "Number of Objects" versus "Collection Size (MB)" or directly the disk on $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/s_SA-ITO* files, but it may be hard to find which one is mapped to which collection. If you have an empty collection (zero objects), but a large collection on disk, a one time workaround could be to clear the kvstore collection ./splunk clean kvstore -app SA-ITOA -collection itsi_refresh_queue ... View more

If you know your events are lagging more than that always than yeah you can use the earliest and latest in your query. But if your data is lagging more than that then you need to once check your data source. Ideally, it should not lag longer. ... View more

Having a lot of jobs artifacts in the dispatch folder, is directly proportional to the number of search jobs, and the time to live of the search artifact (dispatch.ttl) ad-hoc search jobs artifacts usually stay in the dispatch 10 minutes. scheduled search jobs stay for 2 times the scheduling interval (a job running every 4 hours stay on the dispatch for 8 hours) scheduled search jobs that triggered an alert action ( like tracking or alert email) will stay in dispatch for 24h. search jobs shares will be kept for 7 days. realtime searches stay as long at the search is running Knowing that you can look at your dispatch folder and figure what constitutes the mass of job artifacts. https://docs.splunk.com/Documentation/Splunk/latest/Search/Dispatchdirectoryandsearchartifacts Options to reduce the TTL are : For searches - edit the dispatch.ttl in savedsearch.conf. For a particular search, or in the generic settings. - do the same on a per search basis using the UI > searches&reports > advanced edit for alerts - reduce the ttl in alert_actions.conf - or reduce the number of unnecessary alerts. ... View more

Sorry, it's on a different system. Is it possible that there are performance limits in rendering the sparklines for some reason? What I am also learning, I think, is that the sparklines reflect values that correspond to info_min_time and info_max_time in the itsi_summary index, NOT the time that the KPI was run. So, if I have a KPI that looks at data from 45-60 minutes ago, the data point appears at a location on that sparkline that seems to match one of those two times, not the KPI run time. I imagine this contributes to the problem a little as well. ... View more

@esnyder. Thanks for the response. We've since upgraded to 4.1.2 and look forward to 4.4, but the issue still appears to remain. Specifically, when you set up recurring service imports (as upserts), the KPI and Service Id's appear to get regenerated each time the service import is completed. To test this, we had created a glass table based off a variety of KPIs for a service we knew would get 'upserted' in the next run. The service itself did not get duplicated (implying an upsert). However, the glass table reverted all KPIs to ad-hoc searches, the IDs for the service KPIs in the itsi_summary index were different, and the kpi history (in SA sparklines) gets zero'd out back to the most recent run. Could this simply be a bug? ... View more

If I recall correctly, the service analyzer is computing values based off the raw data and not the itsi_summary index. The data in the itsi_summary index will always lag behind the raw data because the summary index doesn't populate instantaneously like the raw data does. So you may have different values depending on how frequently your KPI's they update and their look back period. How far off is the value from the service analyzer view? Have you attempted to duplicate the service and KPI to test if you were getting differing values? ... View more