I saw this one of the monitoring console dashboard and I have been wracking my brain trying to figure out how it gets done.      By "it" I mean the border around the table rows and the bar inside.  I looked at the MC dashboard for it, but I don't see anything special about the search or dashboard (no js or other formatting tricks) that I could discern.       ... View more

If you read the title, you are going "well of course it does", but hear me out.   (This will be a long explanation that will hopefully answer the immediate questions)... Background: We have some on-prem UFs that forward "everything"  to our on-prem enterprise indexers AND specific logs  to our splunk cloud instance indexer.    In case you are wondering,  the cloud instance is where our customer can look at their data without needing access to our internal systems. Problem: Splunk did some maintenance on our cloud instance and when they did so, forwarding  from the UFs also stopped coming into our on-prem Splunk.    I can't figure out why cloud being down would stop the forwarders from sending to enterprise.     Checking the documentation here: https://docs.splunk.com/Documentation/Splunk/8.1.0/Forwarding/Setuploadbalancingd#Configure_universal_forwarder_load_balancing_for_horizontal_scaling It reads like the UFs should switch to the next indexers when it goes down.  But it didn't.  Instead we saw this in the internal logs when the cloud instance was taken down for maintenance  11-25-2020 21:59:48.139 -0600 WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to output group splunkcloud has been blocked for 1200 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.   Looking at the inputs.conf and outputs.conf,  I can see nothing wrong with them to have the data blocked from these UFs Sanitized inputs.conf, with the log that gets sent to both the on-prem instance  (PP_indexers) and cloud instance  bolded  [monitor://C:\blahblahblah\q2.log] _TCP_ROUTING = pp_indexers index = fsd sourcetype = q2 [monitor://C:\blahblahblah\wrapper.log] _TCP_ROUTING = pp_indexers index = fsd_sandbox sourcetype = wrapper [monitor://C:\blahblahblah\metrics.log] _TCP_ROUTING = pp_indexers,splunkcloud index = fsd_sandbox sourcetype = metrics Sanitized outputs.conf:  defaultGroup = pp_indexers forceTimebasedAutoLB = true autoLBFrequency = 15 [tcpout:pp_indexers] server = indexer1.ip.address.here:9997, indexer2.ip.address.here:9997 [tcpout:splunkcloud] compressed = false disabled = false server = our_domain_name.cloud.splunk.com:9997 sslCommonNameToCheck = our_domain_name.cloud.splunk.com sslCertPath = $SPLUNK_HOME/etc/apps/sanitized/client.pem sslPassword = sanitized sslRootCAPath = $SPLUNK_HOME/etc/apps/sanitized/cacert.pem sslVerifyServerCert = true useACK = true Oh and just in case you need it... UF versions are 7.1.2 and 7.2.3 enterprise version is 7.3.4,  cloud is 7.3. ... View more

When I run a report of roles that have been modified in our splunk cloud instance I see these entries Date Time user action info role_name 04-06-20 16:50:18 _ops_admin edit_user granted _ops_admin 04-06-20 16:50:15 _ops_admin edit_user granted _ops_admin 04-06-20 16:49:52 _ops_admin edit_user granted _ops_admin 04-06-20 16:44:06 _ops_admin edit_user granted _ops_admin 04-06-20 16:44:03 _ops_admin edit_user granted _ops_admin We dont have that user or role name defined in our instance. Anyone have an idea of what this is and if I should be concerned? ... View more

Dec 24th is the biggest day for us. During the afternoon, a summary job that populates a summary index for sales reports got delayed due to all of the searches that were happening. Consequently the job ran twice during the next hour. So now I have doubled up sales and transaction amounts for that hour. Is there a way to find and back out the double entries from the summary index? ... View more

A little background: We have a single instance install now and are moving to a multisite index cluster and multisite SH cluster. Our dashboards on the current single instance are 95% inline searches, with a few summary index panels and saved searches in the mix. Very small amount of those. No real-time searches. The dashboards have on average 15 panels that are refreshed every 2 minutes (and yes we need it refreshed that quickly). I have been thinking of rewriting the searches to be base and post-processes. However as I have been reading about the changes needed in the dashboards when in the new environment, I have read some comments and articles where post-processing may not be suitable for multi-index sites. Doing that might be slower than saved searches. Does anybody have any real-world experiences that they can share? I don't want to proceed down one path when the other might be better in the long run. ... View more