Dashboards & Visualizations

After upgrade to 7.1: Bug - Dashboard Chart - drawn pixels stayed incorrectly in old positions even after the Y-axis scale has changed

patng_nw
Communicator

After I upgraded to v7.1, I found some charts in some dashboards are showing incorrect result. I have captured two images to explain what I mean. (same problem in 7.1.1)

Image 1 - During drawing: the chart shows these two lines segments (note: the plotted values are correct) while the dashboard is busy generating all the other data.
alt text

Image 2 - Finish drawing: After all other data points are available, we can see that the scales on the Y-axis has changed (look at those scales in Image 1), and most data points were plotted correctly. However, those data points shown in the Image 1 stayed in their old positions. They should be adjusted based on the new Y-axis scales but they weren't!
alt text

Refreshing the chart inside the dashboard didn't help. If I plot this same chart separately (as a separate search in a new search window), then it displays correctly. Please note that I have double checked the value in tabular format. In tabular format, the values for those data points in Image 1 are correct, but they're positioned incorrectly in the image 2 (their "plotted values" in image 2 are doubled, wrongly).

I didn't remember seeing this problem in 7.0.x. And right after we upgraded to 7.1 and this problem was reported by a user who looked at this dashboard daily. So I think it's introduced by 7.1.

Below is the simplified version of my query:

index="abc_network" sourcetype="abc_network_csv"| append [
| inputcsv abcNetwork_sheetapi.csv 
| eval _time = strptime(date, "%d-%m-%Y")
| where _time>=if(isnull(relative_time(now(),"-30d@d")), "-30d@d", relative_time(now(),"-30d@d")) 
  and _time<if(isnull(relative_time(now(),"-0d@d")), "-0d@d", relative_time(now(),"-0d@d")) 
| rename source as extracted_source 
| ]
| ...(note: some business logic code truncated)...
| timechart span=1day sum(counter1) as Counter1 sum(counter2) as Counter2 by ps

Note: Similar problem happened to other charts (e.g. a barchart) on the same dashboard which which uses similar base queries.

[Update - 2018-06-27]
The problem has been "resolved" by a workaround. In the dashboard we were using base searches (which returns raw events). The workaround is to not use the base searches and instead run it directly in each chart, and that avoided the bug. The case is closed now.

[Update - 2018-07-15]
The problem actually is still here, as I just saw it today, but just less likely to happen after we've applied the "workaround".

0 Karma
1 Solution

MuS
SplunkTrust
SplunkTrust

After reading the update note:

this would most likely still work with a base search if you would use the base search correct 😉 Instead of retuning raw events in the base search simply use a | stats values(*) AS * by _time as last statement or what ever transforming search fits your needs 😉
Read more about base searches, especially why you should not return raw events, here : http://docs.splunk.com/Documentation/Splunk/latest/Viz/Savedsearches#Best_practices

cheers, MuS

View solution in original post

MuS
SplunkTrust
SplunkTrust

After reading the update note:

this would most likely still work with a base search if you would use the base search correct 😉 Instead of retuning raw events in the base search simply use a | stats values(*) AS * by _time as last statement or what ever transforming search fits your needs 😉
Read more about base searches, especially why you should not return raw events, here : http://docs.splunk.com/Documentation/Splunk/latest/Viz/Savedsearches#Best_practices

cheers, MuS

patng_nw
Communicator

@MuS Thanks for the info!

0 Karma

patng_nw
Communicator

The workaround actually just made the problem less likely to happen. Nevertheless it happened again today even after I've removed the base searches.

0 Karma

MuS
SplunkTrust
SplunkTrust

You are welcome!

Converted to an answer, and you are welcome (again 😉 ) to accept it.

cheers, MuS

0 Karma

niketn
Legend

Thanks @MuS, Post Processing Best Practices is where I was trying to go. 🙂

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

patng_nw
Communicator

@niketnilay Please see my latest update above. There is a workaround to this problem.

0 Karma

niketn
Legend

@patng_nw, so was the base search returning raw events or stats? Was it related to performance of base search?

In any case please add details of the Workaround as an answer to your question and accept the same to mark this question as answered!

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

patng_nw
Communicator

@niketnilay It was returning raw events. I just modified the [update] section of the above post. But I have yet to figure out how to mark my original post as answer.

0 Karma

patng_nw
Communicator

@niketnilay: So far this is the response from Splunk technical support:
"Splunk Enterprise 7.1.x is our new product with a lot of UI core level changes, due to which there are many bugs. We are in the process of fixing them in our future releases. A lot of UI related bugs were fixed in 7.1.1 and you could try upgrading to it.

However, I would suggest you roll back to 7.0.x if possible as the possibility of you running into another bug in 7.1.1 is very high"

However, 7.1.1 didn't solve my problem, and Splunk didn't officially support downgrading to 7.0.x either. So it's pretty lame and I am still trying to get them really fix the bug. 😞

0 Karma

niketn
Legend

@patng_nw does the issue also occur when the number or results plotted are less? I mean could this be performance related as you have millions of events plotted across your chart? I wish I could help but I have not been able to recreate the issue! Keep working with Support hopefully they figure it out soon enough!

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

niketn
Legend

@patng_nw, I would think the chart plotted is correct. You seem to have two series Counter1 and Counter2 and Counter2 is setup as Overlay Field with View As Axis option turned on. This makes upto ~150M count on Counter2 plotted in the same chart with equivalent Counter1 upto ~200000 (I would prefer Number Abbreviation turned on for Y-axis as well just for consistency).

You can turn off View as axis to see original count of both Counters on single y-axis. However, I feel the graph is correct, considering you are using overlay. Also validate the Data in Tabular format to double check.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

patng_nw
Communicator

@niketnilay As I explained above, the chart plots correctly if I plot it just by itself (instead of as part of dashboard), and I have already double checked the actual values in tabular format. Therefore the chart is incorrect.

0 Karma

niketn
Legend

@patng_nw, if that is the case, I would suggest you to open a support ticket with Splunk and have them look at this issue. Also add a bug tag to your question.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

patng_nw
Communicator

@niketnilay Thanks. A support ticket was opened.

0 Karma

niketn
Legend

@patng_nw, thanks! Do keep up posted and once you have the resolution to this issue, kindly post here and accept as answer to benefit others facing same issue!

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...