I have a real-time table in one of my dashboards that doesn't update when you first load the page. If you do something with the table - e.g. click one of the fields to sort, it becomes real-time and starts updating in real-time. I'd like to have it be in real-time when you open the dashboard, and would rather not have to instruct everyone who uses the dashboard that you have to sort by a field in order to get it working.
Having a look at http://docs.splunk.com/Documentation/Splunk/7.1.3/Search/Specifyrealtimewindowsinyoursearch shows the setting default_backfill in limits.conf might have an impact on this behaviour.
Also if you leave the dashboard alone for 5 minutes does it then start real-time updating?
Looks like we're closing in on the answer. However, default_backfill is already set to true (and experimentally setting it to false in etc/system/local/limits.conf leads to the same behavior). I wonder if another setting in this file is the solution?
And yes, after roughly 5 minutes it does start updating, which makes sense in accordance with:
For example, if your sliding window is 5 minutes, you will not start to see data until after the first 5 minutes have passed.
The solution the documentation provides is
You can override this behavior so that Splunk software backfills the initial window with historical data before running in the normal real-time search mode.
but default_backfill is already set to true. So I'm not sure how to proceed.
Double check the default_backfill setting with btool http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Usebtooltotroubleshootconfigurati... if it says true then create a self contained app (with problematic dashboard with self contained SPL) and send to support. ( so it is easier for support to remotely test this as well ). It feels like this setting isn't being honoured.
./splunk cmd btool limits list
does list it as true, and
./splunk cmd btool --app=myapp limits list
returns nothing, which makes sense, since there is no local or default limits.conf file in my app directory. So it should default to the system limits.conf, which is indeed set to true according to btool.
I guess the next step is to contact support.
Also, setting default_backfill to false in etc/system/local/limits.conf and running btool says that the value has been set to false. But then the dashboard still backfills 5 minutes (and still doesn't update until you sort it)
So.... it seems the dashboard for some reason knows to backfill 5 minutes, without having read etc/system/local/limits.conf... but when it backfills 5 minutes, it just doesn't update in real-time until you click sort.
My code, in case you'd like to take a look:
<form> <label>title</label> <row> <panel> <title>table title</title> <table> <search> <!-- comment --> <query> index=myindex sender_domain != "abc.com" event_id="DELIVER" | lookup "mycsv.csv" sender | where isnull(precedent) | mvexpand recipient | dedup sender message_subject recipient | where recipient_domain=="abc.com" | eventstats count as recipient_count by sender,message_subect | dedup sender,message_subject | sort - time | table sender message_subject recipient_count _time </query> <earliest>rt-5m</earliest> <latest>rt</latest> </search> <option name="wrap">true</option> <option name="rowNumbers">false</option> <option name="drilldown">cell</option> <option name="dataOverlayMode">none</option> <option name="count">25</option> </table> </panel> </row> </form>