Dashboards & Visualizations

Dasboard Panel Not Getting New Events

SplunkTrust
SplunkTrust

I have a script that gathers disk freespace metrics and indexes these metrics into splunk every 10 minutes.

This is the dashboard panel I am trying to get to work.

<module name="HiddenSearch" layoutPanel="panel_row2_col1" group="LVM2 Filesystem Capacity" autoRun="True">
<param name="search">sourcetype=scriptlog name=mon-lvm2 | dedup host,filesystem | sort capacity desc | fields host,filesystem,capacity</param>
<param name="latest">rt</param>
<param name="earliest">rt</param>
<module name="EnablePreview">
   <param name="enable">true</param>
   <param name="display">false</param>
<module name="SimpleResultsTable">
 <param name="dataOverlayMode">heatmap</param>
</module>

When I navigate to the dashboard, it hangs at "Waiting for search events..." until the next 10 minute interval. Once this 10 minute interval is reached, it will fill the panel with the proper events. After this it will not update with any new events.

Why does it immediately find the latest events? I.E. I open dashboard at 35 after, and it has to wait until 40 after before it fills the panel. Why doesn't it find the events at 30 after, which is what the simple search would do. When it does fill the table, at 40 after, how come it doesn't update with new events at 50 after?

0 Karma
1 Solution

SplunkTrust
SplunkTrust

-- Real time searches right now dont search the index at all. Instead the real time search gets its events right from the point where the events are handed off from all the data inputs to the indexer. This confuses people all the time so you're not alone.

I'm pretty sure that a future release will improve that behavior to the point where you will have both -- the old events out of the index (back to whatever the window size is) will be merged with the new events fresh off the wire and you'll be presented with results that make sense.

-- As to the 'rt' and real time windowing: if you run a search with both earliest and latest set to 'rt', then that's an unbounded or unwindowed real time search. For example if you were running that search and suddenly indexed some old logs from 2007, they'd show up in your search. (Your timeline would jump wildly out to show 2007 through 2010)

-- And about why it seemed to only be updating once, I just ran a quick test with dedup and i see the same thing on 4.1.5. I run index=_internal source=*metrics.log | dedup group series, as an unbounded real time search. And based on its behavior with historical searches I'd expect the most recent _raw to always be reflected. However it's the first ie the oldest _raw values that it always shows.

I think its a bug where dedup is following its mission but should probably modify that mission when it notices itself running in realtime--- For any given field value, dedup will keep the "first value" it sees. In historical searches where the cursor is moving backwards in time through the events, this is great. In a real time search where the cursor moves forward, it's not what you want.

View solution in original post

SplunkTrust
SplunkTrust

-- Real time searches right now dont search the index at all. Instead the real time search gets its events right from the point where the events are handed off from all the data inputs to the indexer. This confuses people all the time so you're not alone.

I'm pretty sure that a future release will improve that behavior to the point where you will have both -- the old events out of the index (back to whatever the window size is) will be merged with the new events fresh off the wire and you'll be presented with results that make sense.

-- As to the 'rt' and real time windowing: if you run a search with both earliest and latest set to 'rt', then that's an unbounded or unwindowed real time search. For example if you were running that search and suddenly indexed some old logs from 2007, they'd show up in your search. (Your timeline would jump wildly out to show 2007 through 2010)

-- And about why it seemed to only be updating once, I just ran a quick test with dedup and i see the same thing on 4.1.5. I run index=_internal source=*metrics.log | dedup group series, as an unbounded real time search. And based on its behavior with historical searches I'd expect the most recent _raw to always be reflected. However it's the first ie the oldest _raw values that it always shows.

I think its a bug where dedup is following its mission but should probably modify that mission when it notices itself running in realtime--- For any given field value, dedup will keep the "first value" it sees. In historical searches where the cursor is moving backwards in time through the events, this is great. In a real time search where the cursor moves forward, it's not what you want.

View solution in original post

SplunkTrust
SplunkTrust

thanks for the insight nick

0 Karma

SplunkTrust
SplunkTrust

The problem at least in some part comes from the "dedup host,filesystem" part of the search string. I took that out and was able to get it to update, but when it did it included the old events. I changed the "window" from earliest=rt latest=rt to earliest=rt-10m to latest=rt

This matched up with the interval that the script is run on, so it will only present one set of disk metrics, which is what I want.

The rt dashboard stuff seems to work by constantly search for events that satisfy the supplied search string. I suppose if someone wants to supply it a better answer, it would include a rational for the screwy behavior I was seeing by have the dedup function included in the search string.

0 Karma

SplunkTrust
SplunkTrust

a preemptive apology for the confusing chronological mess

0 Karma