Splunk Search

Splunk removes all items from chart apart from OTHER

bowesmana
SplunkTrust
SplunkTrust

I have a timechart that shows

timechart span=5s fixedrange=f limit=20 perc90(time_taken) by request

over 70 minutes. Used for performance testing.

It's in a real time dashboard and it shows perfectly well. However, I redraw the complete dashboard to cover the 70 minutes at the end of a test and while the chart fills in, just before the end of drawing all events, all the lines in the chart disappear, leaving only the metrics for OTHER...???

I have opened up the chart in its own search and it will do the same

However, this does not always happen for different runs of the same test, but appears consistent with some test runs.

There are around 18 million events.
I changed the limit= to 15, then 5, but still the same problem. I then changed the limit back to 20 and the span from 5s to 1m and it works.

I guess it's a limits thing, but the odd thing is that it never does this for a similar timechart that has a limit=28 on the same data, but is just doing the same timechart, but instead of perc90, is doing

count by request

samjenk_2
Explorer

I'm having a similar issue on a search head running 6.6.4, but not on 6.4.7. Can you report your version of Splunk you're using?

When searching over large data sets (2.2 million events) and using a | timechart count by <fieldname> clause, 6.4.7 returns a table and chart of the top ten values of <fieldname> plus the rest aggregated as OTHER. This is the expected behavior.

Running the same search, Splunk 6.6.4 proceeds normally, graphing the top 10 + OTHER, until just before the search finalizes. When the search finalizes, all of the columns are aggregated into OTHER in both the Statistics and Visualization tabs. This is not expected, and is repeatable on searches returning more than 2.2 million events. Both of these search heads are searching over the same data. When timechart is invoked on 6.6.4 with useother=f, the search returns no results. Fore smaller data sets (fewer than 2.2 million events), 6.6.4 behaves as 6.4.7 does.

cmerriman
Super Champion

70 minutes * 5 seconds is only 350 data points per line, so that shouldn't be an issue.
http://docs.splunk.com/Documentation/Splunk/6.6.3/Viz/ChartDisplayissues

have you tried to set useother=f

0 Karma

bowesmana
SplunkTrust
SplunkTrust

Not tried useother, but I want other. Odd thing is that it can display a 75 minute rt dashboard and when redrawing the dashboard as last 70, it shows the count chart correctly with 28 items at 5s, but not the percentile chat with 20 items at 5s

0 Karma

DalJeanis
Legend

@cmerriman - there are 12 5-second intervals in a minute. 70*12=840 >800. Limit for 5 seconds would be 4000 seconds, about 66 minutes.

cmerriman
Super Champion

good catch @DalJeanis, multiplied when i should have divided....i thought that sounded wrong, but couldn't put my finger on it....

Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...