Kindly, check for which specific indexes and for which bucket directories it is giving the error. Generally, whenever an index generates too many small tsidx files(more than 25) Splunk is not able to optimize all those files within the specified time period. Kindly, run the below command against the specific directory to optimize it manually:- splunk-optimize -d|--directory Or you can make the below changes in Indexes.conf to fix the issue:- indexes.conf [default] maxConcurrentOptimizes=25 maxRunningProcessGroups=12 processTrackerServiceInterval=0 Please go through the below documentation to have a better understanding of Splunk Optimization. http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Optimizeindexes ... View more

You need to provide in total three capabilities to view the monitoring console for user privileges. 1).admin_all_objests. 2).dispatch_rest_to_indexers. 3).edit_dist_peer. The dispatch_rest_to_indexers capability will show the resource usage of each instance and edit_dist_peer will fix the instance unreachable error. ... View more

$job.earliestTime$ and $job.latestTime$ are both job information tokens and they do not work with Dashboards Scheduled PDF Delivery.If you schedule a PDF Report it works but with Dashboards Job information tokens doesn't work. The above tokens gives information about specific search jobs and a Dashboard is made up of multiple Reports which in turn has multiple searches.So it cannot extract specific search job information. Because of which we are getting some weird results. The below document clearly states Job information tokens are not supported for Scheduled PDF Delivery for Dashboards. http://docs.splunk.com/Documentation/Splunk/6.5.2/Viz/DashboardPDFs#Tokens_available_for_email_notifications ... View more

$job.earliestTime$ and $job.latestTime$ are both job information tokens and they do not work with Dashboards Scheduled PDF Delivery.If you schedule a PDF Report it works but with Dashboards Job information tokens doesn't work. The above tokens gives information about specific search jobs and a Dashboard is made up of multiple Reports which in turn has multiple searches.So it cannot extract specific search job information. Because of which we are getting some weird results. The below document clearly states Job information tokens are not supported for Scheduled PDF Delivery for Dashboards. http://docs.splunk.com/Documentation/Splunk/6.5.2/Viz/DashboardPDFs#Tokens_available_for_email_notifications ... View more

This error occurs when your Search Heads attempts to send a search job to a Search Peer (usually one of your Indexers) and the Indexer does not respond in within the default timeout period so the Search continues but without using that Indexer (which of course probably means that some of your events are not returned so your search is wrong). In my experience, the problem can often be cleared simply by restarting the Splunk instance on the Indexer in question but sometimes you need to dig deeper. In any case, something is keeping your Indexers so busy that it cannot reliably respond to search requests even though the Splunk instance is running. I am sure this kind of thing can also commonly be caused by misconfigured/misbehaving load-balancers or other identity/load-shifting equipment that is between your Search Head and your Indexer peers. ... View more

Generally, you will get the error If the account you are using to log in to the instance doesn't have the dispatch_rest_to_indexers capability. You need to add the Dispatch_rest_to_indexers capability to the respective role or the user to make it work. Or you can add it to the default stanza in authorize.conf so that everyone has that capability. [default] dispatch_rest_to_indexers = enabled ... View more

If you are getting the error only for this specific directory you can run Splunk Optimization manually by the below command:- splunk-optimize -d|--directory But If you are getting multiple errors for different directories then It might be a bug. In that case First, check what is the Splunk Version you are running on the Server. It is a bug in older versions of Splunk(till 7.0.4.) and was fixed from 7.0.5.onwards. If it is a lower version of Splunk you can upgrade Splunk to a higher version or you can make the below changes in your Indexes.conf to fix the issue. indexes.conf [default] maxConcurrentOptimizes=25 maxRunningProcessGroups=12 processTrackerServiceInterval=0 ... View more

Kindly, check for which specific indexes and for which bucket directories it is giving the error. Generally, whenever an index generates too many small tsidx files(more than 25) Splunk is not able to optimize all those files within the specified time period. Kindly, run the below command against the specific directory to optimize it manually:- splunk-optimize -d|--directory Or you can make the below changes in Indexes.conf to fix the issue:- indexes.conf [default] maxConcurrentOptimizes=25 maxRunningProcessGroups=12 processTrackerServiceInterval=0 Please go through the below documentation to have a better understanding of Splunk Optimization. http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Optimizeindexes ... View more

As you told that you have disabled and deleted those two apps but still error shows that these two apps are causing accelerated data models. First check whether the searches are still coming from those two apps.if it still coming from those two apps it means the apps are not properly deleted or maybe the apps are present somewhere else in your environment which is also forwarding data from those two apps. ... View more

For migrating the historical data from an index from old to new cluster you need to follow the below steps:- Put CM in Maintenance mode before performing the below steps. stop Splunk on the indexers (old existing cluster). copy the original buckets (not replicated) manually from the existing cluster to a different location (e.g./tmp) on the same indexers. create the index on the new indexer cluster master and push it to the new indexers. update the GUID of the new indexer cluster peers in the old indexers bucket id. (i.e. replace the current indexers GUID with the new indexers GUID at the end of the bucket name) place the buckets copied from the existing setup to the new cluster with respective bucket numbers. (bucket numbers should not conflict with each other) start indexers for the bucket to replicate to the other indexers. To get the GUID of the indexers, you can either use CM's indexer clustering page in the GUI and expand the indexer details in peers tab or navigate to the CLI of the respective indexers and check $SPLUNK_HOME/etc/instance.cfg file. ... View more

You need to provide in total three capabilities to view the monitoring console for user privileges. 1).admin_all_objests. 2).dispatch_rest_to_indexers. 3).edit_dist_peer. The dispatch_rest_to_indexers capability will show the resource usage of each instance and edit_dist_peer will fix the instance unreachable error. ... View more

In Splunk 7.2.x opening an external link in a new tab from the app navigation bar works. Example data/ui/nav/default.xml file including a link to the Splunk Developer Portal: Splunk Developer Portal Please Note:-It works only for 7.2.x versions and above.Tried it with lower versions(for ex.6.5.x) of splunk and it's not working.For 7.2.x it works for pre-defined apps as well as custom apps also. ... View more