After we upgraded from 8.0.7 to 8.2.3, we are having lots of problems with search performance.  We noticed that the analytics workspace changed a great deal and we wonder if that could be causing the performance issue.   Now we have lots of searches queued - that didn't happen before.  Also sometimes the maximum number of historical searches is exceeded and we end up having to restart Splunk.  After the restart, our system runs okay for a while. Any help will be appreciated. ... View more

We have an email set to run on a schedule from a dashboard.   It worked yesterday but not today.  I can see in the log where the email is sending.  I tested by changing the scheduled time and again, it says the email is sending, but it does not arrive.   Another email that comes from a scheduled report but it did not arrive today either. ... View more

Is there a way in the Splunk Metrics Workshop to customize the height of the charts? ... View more

I have a forwarder that forwards to two different Splunk systems: SplunkA and SplunkB. The data coming into the forwarder is TCP data on two different ports, port A and port B. All data on port A is forwarded to SplunkA and all data on port B is forwarded to SplunkB. I configured persistent queuing on both ports. Then I shutdown SplunkA. I found that both both port A and port B data was queued. That means that none of the TCP data on port B was getting to SplunkB, even though SplunkB was up. Is this expected behavior? ... View more

I am using a multiselect input to create a query of the metrics store. If there is one metric_name chosen, the resulting token looks like this: (metric_name="myMetricName") If two metric_names are chosen the resulting token looks like this: (metric_name="myMetricName1" OR metric_name="myMetricName2") The search looks like this: |mstats prestats=true avg(_value) WHERE $myToken$ index="myMetricsIndex" span=1m BY metric_name |timechart span=1m avg(_value) by metric_name This search works for two metric_names but not for one metric_name. I have two questions. First: Why is the following search illegal? |mstats prestats=true avg(_value) WHERE (metric_name="myMetricName") index="myMetricsIndex" span=1m BY metric_name |timechart span=1m avg(_value) by metric_name Secondly: How to I fix my multiselect so that it works for one metric tag? ... View more

Currently we have a standalone Splunk instance. All of the data that is indexed comes as UDP data over a data diode. We want to move to one search head and three indexers. In order to have load balancing on the indexers, we plan to have all of the data from the data diode go to a server with a heavy forwarder installed. The heavy forwarder will balance the load to the Splunk indexers. The heavy forwarder and its server become a single point of failure. Questions: Is there a better way to set this up with a data diode? How do we size the server with the heavy forwarder? How can we "harden" the heavy forwarder to avoid down time? ... View more

I am trying to pull incoming tcp data into the Metrics Store using this information: http://docs.splunk.com/Documentation/Splunk/7.0.0/Metrics/GetMetricsInOther#Get_metrics_in_from_clients_over_TCP.2FUDP Currently the data is coming in to an event index and the _raw field looks like this: 2017-11-29 18:43:22.807 +0000 Tag="Channel1.Device1.Tag1" Value="1523" Quality="good" We would like to pull this into the Metrics Store with these fields metric_name Tag _value Value _time _time (We also want the Quality, but I'm ignoring that until I get the basics.) Created a new port for the data to come in. Source Sourcexxxxx Source Type Typexxxxx index indexxxxxx The index (indexxxxx) is a metrics index I edit configuration files as below and restart Splunk. I get two errors: Index Processor: Metric value= is not valid for source=Sourcexxxxx, sourcetype=Typexxxxx, host=Hostxxxxx, index=index51115. Metric event data with an invalid metric value would not be indexed. Ensure the input metric data is not malformed. Index Processor: Metric name is missing for source=Sourcexxxxx, sourcetype=Typexxxxx, host=Hostxxxxx, index=indexxxxxx. Metric event data without metric name is invalid and would not be indexed. Ensure the input metric data is not malformed. What am I doing wrong? Thanks in advance for your help. props.conf [Typexxxxx] TIME_FORMAT = %F %H:%M:%S.%Q %z TRANSFORMS-test_metric_name = test_name TRANSFORMS-test_metric_value = test_value NO_BINARY_CHECK = true SHOULD_LINEMERGE = false pulldown_type = 1 category = Metrics transforms.conf [test_name] REGEX = "Channel1.Device1.(\S+)" FORMAT = metric_name::$1 WRITE_META = true [test_value] REGEX = Value="(\S+)" FORMAT = _value::$1 WRITE_META = true fields.conf [metric_name] INDEXED=true [_value] INDEXED=true ... View more

For my search for the horizon chart includes: |timechart avg(R) as R by T. I have 18 values of T. However, the horizon chart seems to be limited to 10 rows. Is there an option to change the number of rows? ... View more