topic Re: After upgrade to 7.1: Bug - Dashboard Chart - drawn pixels stayed incorrectly in old positions even after the Y-axis scale has changed in Dashboards & Visualizations

After upgrade to 7.1: Bug - Dashboard Chart - drawn pixels stayed incorrectly in old positions even after the Y-axis scale has changed

patng_nw — Thu, 24 May 2018 04:04:50 GMT

After I upgraded to v7.1, I found some charts in some dashboards are showing incorrect result. I have captured two images to explain what I mean. (same problem in 7.1.1)

Image 1 - During drawing: the chart shows these two lines segments (note: the plotted values are correct) while the dashboard is busy generating all the other data.

Image 2 - Finish drawing: After all other data points are available, we can see that the scales on the Y-axis has changed (look at those scales in Image 1), and most data points were plotted correctly. However, those data points shown in the Image 1 stayed in their old positions. They should be adjusted based on the new Y-axis scales but they weren't!

Refreshing the chart inside the dashboard didn't help. If I plot this same chart separately (as a separate search in a new search window), then it displays correctly. Please note that I have double checked the value in tabular format. In tabular format, the values for those data points in Image 1 are correct, but they're positioned incorrectly in the image 2 (their "plotted values" in image 2 are doubled, wrongly).

I didn't remember seeing this problem in 7.0.x. And right after we upgraded to 7.1 and this problem was reported by a user who looked at this dashboard daily. So I think it's introduced by 7.1.

Below is the simplified version of my query:

index="abc_network" sourcetype="abc_network_csv"| append [
| inputcsv abcNetwork_sheetapi.csv 
| eval _time = strptime(date, "%d-%m-%Y")
| where _time>=if(isnull(relative_time(now(),"-30d@d")), "-30d@d", relative_time(now(),"-30d@d")) 
  and _time<if(isnull(relative_time(now(),"-0d@d")), "-0d@d", relative_time(now(),"-0d@d")) 
| rename source as extracted_source 
| ]
| ...(note: some business logic code truncated)...
| timechart span=1day sum(counter1) as Counter1 sum(counter2) as Counter2 by ps

Note: Similar problem happened to other charts (e.g. a barchart) on the same dashboard which which uses similar base queries.

[Update - 2018-06-27]
The problem has been "resolved" by a workaround. In the dashboard we were using base searches (which returns raw events). The workaround is to not use the base searches and instead run it directly in each chart, and that avoided the bug. The case is closed now.

[Update - 2018-07-15]
The problem actually is still here, as I just saw it today, but just less likely to happen after we've applied the "workaround".

Re: After upgrade to 7.1: Bug - Dashboard Chart - drawn pixels stayed incorrectly in old positions even after the Y-axis scale has changed

niketn — Thu, 24 May 2018 08:11:59 GMT

@patng_nw, I would think the chart plotted is correct. You seem to have two series Counter1 and Counter2 and Counter2 is setup as Overlay Field with View As Axis option turned on. This makes upto ~150M count on Counter2 plotted in the same chart with equivalent Counter1 upto ~200000 (I would prefer Number Abbreviation turned on for Y-axis as well just for consistency).

You can turn off View as axis to see original count of both Counters on single y-axis. However, I feel the graph is correct, considering you are using overlay. Also validate the Data in Tabular format to double check.

Re: After upgrade to 7.1: Bug - Dashboard Chart - drawn pixels stayed incorrectly in old positions even after the Y-axis scale has changed

patng_nw — Thu, 24 May 2018 08:20:06 GMT

@niketnilay As I explained above, the chart plots correctly if I plot it just by itself (instead of as part of dashboard), and I have already double checked the actual values in tabular format. Therefore the chart is incorrect.

Re: After upgrade to 7.1: Bug - Dashboard Chart - drawn pixels stayed incorrectly in old positions even after the Y-axis scale has changed

niketn — Fri, 25 May 2018 04:57:25 GMT

@patng_nw, if that is the case, I would suggest you to open a support ticket with Splunk and have them look at this issue. Also add a bug tag to your question.

Re: After upgrade to 7.1: Bug - Dashboard Chart - drawn pixels stayed incorrectly in old positions even after the Y-axis scale has changed

patng_nw — Fri, 25 May 2018 05:20:06 GMT

@niketnilay Thanks. A support ticket was opened.

Re: After upgrade to 7.1: Bug - Dashboard Chart - drawn pixels stayed incorrectly in old positions even after the Y-axis scale has changed

niketn — Fri, 25 May 2018 07:47:10 GMT

@patng_nw, thanks! Do keep up posted and once you have the resolution to this issue, kindly post here and accept as answer to benefit others facing same issue!

Re: After upgrade to 7.1: Bug - Dashboard Chart - drawn pixels stayed incorrectly in old positions even after the Y-axis scale has changed

patng_nw — Fri, 01 Jun 2018 09:51:09 GMT

@niketnilay: So far this is the response from Splunk technical support:
"Splunk Enterprise 7.1.x is our new product with a lot of UI core level changes, due to which there are many bugs. We are in the process of fixing them in our future releases. A lot of UI related bugs were fixed in 7.1.1 and you could try upgrading to it.

However, I would suggest you roll back to 7.0.x if possible as the possibility of you running into another bug in 7.1.1 is very high"

However, 7.1.1 didn't solve my problem, and Splunk didn't officially support downgrading to 7.0.x either. So it's pretty lame and I am still trying to get them really fix the bug. 😞

Re: After upgrade to 7.1: Bug - Dashboard Chart - drawn pixels stayed incorrectly in old positions even after the Y-axis scale has changed

niketn — Fri, 01 Jun 2018 13:40:19 GMT

@patng_nw does the issue also occur when the number or results plotted are less? I mean could this be performance related as you have millions of events plotted across your chart? I wish I could help but I have not been able to recreate the issue! Keep working with Support hopefully they figure it out soon enough!

Re: After upgrade to 7.1: Bug - Dashboard Chart - drawn pixels stayed incorrectly in old positions even after the Y-axis scale has changed

patng_nw — Thu, 28 Jun 2018 03:58:39 GMT

@niketnilay Please see my latest update above. There is a workaround to this problem.

Re: After upgrade to 7.1: Bug - Dashboard Chart - drawn pixels stayed incorrectly in old positions even after the Y-axis scale has changed

niketn — Thu, 28 Jun 2018 07:38:31 GMT

@patng_nw, so was the base search returning raw events or stats? Was it related to performance of base search?

In any case please add details of the Workaround as an answer to your question and accept the same to mark this question as answered!

Re: After upgrade to 7.1: Bug - Dashboard Chart - drawn pixels stayed incorrectly in old positions even after the Y-axis scale has changed

patng_nw — Thu, 28 Jun 2018 08:19:45 GMT

@niketnilay It was returning raw events. I just modified the [update] section of the above post. But I have yet to figure out how to mark my original post as answer.

Re: After upgrade to 7.1: Bug - Dashboard Chart - drawn pixels stayed incorrectly in old positions even after the Y-axis scale has changed

MuS — Thu, 28 Jun 2018 10:11:41 GMT

After reading the update note:

this would most likely still work with a base search if you would use the base search correct 😉 Instead of retuning raw events in the base search simply use a | stats values(*) AS * by _time as last statement or what ever transforming search fits your needs 😉
Read more about base searches, especially why you should not return raw events, here : http://docs.splunk.com/Documentation/Splunk/latest/Viz/Savedsearches#Best_practices

cheers, MuS

Re: After upgrade to 7.1: Bug - Dashboard Chart - drawn pixels stayed incorrectly in old positions even after the Y-axis scale has changed

patng_nw — Thu, 28 Jun 2018 10:18:30 GMT

@MuS Thanks for the info!

Re: After upgrade to 7.1: Bug - Dashboard Chart - drawn pixels stayed incorrectly in old positions even after the Y-axis scale has changed

MuS — Thu, 28 Jun 2018 10:21:35 GMT

You are welcome!

Converted to an answer, and you are welcome (again 😉 ) to accept it.

cheers, MuS

Re: After upgrade to 7.1: Bug - Dashboard Chart - drawn pixels stayed incorrectly in old positions even after the Y-axis scale has changed

niketn — Thu, 28 Jun 2018 10:35:57 GMT

Thanks @MuS, Post Processing Best Practices is where I was trying to go. 🙂

Re: After upgrade to 7.1: Bug - Dashboard Chart - drawn pixels stayed incorrectly in old positions even after the Y-axis scale has changed

patng_nw — Mon, 16 Jul 2018 01:18:10 GMT

The workaround actually just made the problem less likely to happen. Nevertheless it happened again today even after I've removed the base searches.