After we upgraded from 8.0.7 to 8.2.3, we are having lots of problems with search performance.  We noticed that the analytics workspace changed a great deal and we wonder if that could be causing the performance issue.   Now we have lots of searches queued - that didn't happen before.  Also sometimes the maximum number of historical searches is exceeded and we end up having to restart Splunk.  After the restart, our system runs okay for a while. Any help will be appreciated. ... View more

We have an email set to run on a schedule from a dashboard.   It worked yesterday but not today.  I can see in the log where the email is sending.  I tested by changing the scheduled time and again, it says the email is sending, but it does not arrive.   Another email that comes from a scheduled report but it did not arrive today either. ... View more

Is there a way in the Splunk Metrics Workshop to customize the height of the charts? ... View more

I have a forwarder that forwards to two different Splunk systems: SplunkA and SplunkB. The data coming into the forwarder is TCP data on two different ports, port A and port B. All data on port A is forwarded to SplunkA and all data on port B is forwarded to SplunkB. I configured persistent queuing on both ports. Then I shutdown SplunkA. I found that both both port A and port B data was queued. That means that none of the TCP data on port B was getting to SplunkB, even though SplunkB was up. Is this expected behavior? ... View more

I am using a multiselect input to create a query of the metrics store. If there is one metric_name chosen, the resulting token looks like this: (metric_name="myMetricName") If two metric_names are chosen the resulting token looks like this: (metric_name="myMetricName1" OR metric_name="myMetricName2") The search looks like this: |mstats prestats=true avg(_value) WHERE $myToken$ index="myMetricsIndex" span=1m BY metric_name |timechart span=1m avg(_value) by metric_name This search works for two metric_names but not for one metric_name. I have two questions. First: Why is the following search illegal? |mstats prestats=true avg(_value) WHERE (metric_name="myMetricName") index="myMetricsIndex" span=1m BY metric_name |timechart span=1m avg(_value) by metric_name Secondly: How to I fix my multiselect so that it works for one metric tag? ... View more

Currently we have a standalone Splunk instance. All of the data that is indexed comes as UDP data over a data diode. We want to move to one search head and three indexers. In order to have load balancing on the indexers, we plan to have all of the data from the data diode go to a server with a heavy forwarder installed. The heavy forwarder will balance the load to the Splunk indexers. The heavy forwarder and its server become a single point of failure. Questions: Is there a better way to set this up with a data diode? How do we size the server with the heavy forwarder? How can we "harden" the heavy forwarder to avoid down time? ... View more