Splunk Search

Maximum number of events displayed in an "events list" visualization?

Graham_Hanningt
Builder

I have a dashboard in Splunk 7.3.0 with the following HTML viz definition:

<html depends="$eventCount$,$duration$,$startTime$,$endTime$">
    $eventCount$ events spanning $duration$ ($startTime$ to $endTime$)
</html>

The rendered HTML viz output shows numbers that match my expectations, because I know what data has been ingested, and how I'm searching that data:

114658 events spanning 1 hour 10 minutes 57 seconds (08/14/17 06:09:54 to 07:20:51)

The dashboard also contains a timechart command to display a histogram of events. Mousing over the leftmost bar in that histogram shows the expected earliest time, 6:09 AM; the rightmost bar shows the expected latest time, 07:20 AM.

So far, so good.

The dashboard also contains an "event(s) list" ( <event> ) visualization. I expected—I now realize, mistakenly, hence this question—that the events list would show all of the 114658 events.

However, across the 134 pages of that events list viz, the events are only from the very end of the time period, around 07:20 AM.
The events list viz shows 10 events per page: so, across those 134 pages, that's a total of 1340 events.

When I inspect the viz, the inspector reports "This search has completed and has returned 4,213 results by scanning 114,658 events". (Curiously, sometimes, for the same data, the inspector reports exactly 1000 fewer results: "3,213 results".)

When I click the "Open in Search" (magnifying glass) option in the viz, the Search tab shows the expected 114658 events.

I understand that Splunk visualizations can only handle a certain number of data points. (Generally, this is not an issue for me, because I let timechart autospan, or I use the head command to get "top n" results.)

Somehow, though, I thought that events lists were "special"; that they would act just like the results of the default Search app. But apparently not.

Could someone please explain to me, or point me to the relevent Splunk docs that covers, the maximum number of events displayed in an events list viz, and how that relates to the number of results displayed in the inspector? (Because the number of results reported by the inspector doesn't match my arithmetic of number of pages multiplied by number events shown per page.)

That limit doesn't appear to be, exactly, charting.data.count.

(All of the visualizations that I'm referring to in this dashboard have essentially the same base search, the same earliest and latest time.)

My current practice regarding events lists

My typical practice when designing dashboards is to have various custom UI controls that set tokens to filter the search(es) used by the dashboard visualizations. At the bottom of the dashboard, I have an events list, so that users can look at the details of the events shown in the visualizations.

By default, depending on the nature of the data, until the user sets those UI controls to narrow the search results, a search might return many thousands of events (in the example described above, 114658 events).

When I set my UI controls to filter the search results down to a few dozen events list, then, sure, the "problem" I describe in this question evaporates: the events list displays the full set of events, from the earliest to the latest times.

Should I rethink how I use events lists?

I mistakenly expected events lists to be "comprehensive" regardless of the number of events returned by a search. Clearly, I'm wrong about that. So, in a sense, events lists are only useful when the number of events to be displayed in the list is below a certain limit. Beyond that limit, the events list doesn't show all of the events that users might expect, based on the UI controls and other visualizations in the same dashboard.

I'm not sure what to do about this. I like the idea of dashboards that can flexibly scale from visualizing many thousands of events down to a few events. In practice, however, the way that I'm currently implementing the events list viz does not scale well. I'm tempted to look into how to display a "too many events to list" message in place of an events list that isn't comprehensive.

Feedback, suggestions welcome.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...