If you schedule the first search, you could schedule a second search which looks for result_count=0 in the internal log: index=_internal startminutesago=5 sourcetype=scheduler savedsearch_name="name_of_first_scheduled_search" status=success result_count=0 | append [ search "second search query" ] ... View more

It is broken in 5.0.3 still.... I'm getting iis-2 sourcetype despite hardcoding it in inputs.conf ... View more

I have seen this happening when there is a high count of jobs in dispatch. If you are able to, run clean dispatch command to delete jobs older than X days or X hours (choose a number for the days or hours) http://answers.splunk.com/answers/139924/where-can-i-find-documentation-for-splunkd-clean-dispatch-command If you're in a pinch, delete everything in the dispatch directory but be aware that you may have some artifacts used by dashboards or saved by users that will be deleted. The search can always run again but that might not be ideal. ... View more

It was extremely confusing to find two new apps residing in the splunk_app_for_nix: SA-nix and Splunk_TA_nix. Previously SA and TA are installed on indexers in a SH_POOL environment. In this app: "The Splunk Supporting Add-on for Unix and Linux provides search macros, reports, and other configurations for the Splunk App for Unix and Linux. A Splunk App for Unix and Linux instance must have the Supporting Add-on installed in order to function correctly." What the documentation fails to mention is that the SA- and TA are INSTALLED AUTOMATICALLY by the splunk_app_for_nix. This was extremely frustrating to figure out where they were coming from. Please update the documentation -- it would be a problem for organizations who have tightly controlled configuration management to have software installed automatically. ... View more

Looks like in version 5+ you can set the index in wmi.conf file. ... View more

Spent way too much time trying to figure this out and have lost interest in trying to do this. ... View more

You can modify the default.xml to use flashtimeline instead of search app: Instead of using search in default.xml: <nav search_view="search" color="#65A637"> <view name="search" default='true' /> <view name="data_models" /> <view name="reports" /> <view name="alerts" /> <view name="dashboards" /> </nav> Use (I just swapped flashtimeline in place of search) and added another line for dashboard_live. <nav search_view="search" color="#65A637"> <view name="flashtimeline" default='true' /> <view name="dashboard_live" /> <view name="data_models" /> <view name="reports" /> <view name="alerts" /> <view name="dashboards" /> </nav> ... View more

In ~/etc/system/local/ui-prefs.conf on your search head, set the following which will set "Last 15 minutes" as your default search time range in the picker: [default] dispatch.earliest_time = -15m dispatch.latest_time = now ... View more

The documentation apparently doesn't exist as of this date. From what I have been able to figure out: Syntax: ~/splunk/bin/splunk cmd splunkd clean-dispatch /tmp -24h@h /tmp = the directory where you want the dispatch artifacts to be copied to. -24h@h = the age when older dispatch artifacts are moved out of dispatch ... View more

The base_max_searches setting has nothing to do with the srchJobsQuota setting under roles. The former is a system-wide server setting. ... View more

Thanks, Octavio! I will use your response to convince the team that we need to allocate more storage to our SHs. ... View more

Why is anything being written to local in a POOLING scenario? We discovered that this large artifact was caused by SoS app version 3.1 (and prior?), the RT nature of the "Real-time measured indexing rate and latency" panel (the top panel in the Distributed Indexing Performance view) caused an extremely huge dispatch artifact to be created if the user allowed the panel to continue to run. In version 3.2, the rt was removed and a Run button was added in addition to the following disclaimer: Caution: This search can be resource intensive and should not run indefinitely. Use the search controls on the right to cancel, pause, or finalize the search ... View more

Then what do you set it up to do if you do NOT set it to alert every time? As far as I can tell (in version 5), there is no setting to tell Splunk this is not an alert. The alert condition and mode is automatically set for any scheduled search. ... View more

Looks like this is stored in memory so the message will be lost if the server is restarted. ... View more