I was wondering if anyone else was having this problem – most of the dashboards are very poorly populated unless I have my user profile set to system default time.
Let me add a little more background:
I am dealing with a distributed environment where all my servers are set for UTC.
I have the Technology Add-on for Fire Brigade installed on all my indexers. It properly created and updates the “monitored_indexes.csv”
I have the Fire Brigade app installed only on the search head. On each of the dashboards, my host and index drop downs being populated as expected.
The “Retention Overview” dashboard is a good example to highlight my problem. If I have my profile set for system default time, every single table populates. If I switch my profile default time to PST or EST, not a single table populates.
Another example is the “Index Detail” dashboard. If I have my time set for EST/PST, I get the correct dropdowns and the following graphics are populated: Sourcetype Portions, Compression percentage, Compressed Usage vs Raw Volume. Now, if I switch my profile to system default time the whole page is populated.
-Fire Brigade App – 211
-Technology Add-on for Fire Brigade – 204
-Splunk Enterprise 6.5
Is anyone else seeing this issue? Is it a requirement that my profile time must be the same as system time for the API calls to work?
The problem here is that the nightly data collection (done by the TA on the indexers) does so only once a day, just after midnight. The dashboards which focus on "current state" do so by selecting "earliest=@d" events from those collected by the TA. When the servers are in UTC, midnight happens at 4pm Pacific (or 5, depending upon daylight savings), so between midnight Pacific and the afternoon, "today" in Pacific time doesn't have any records. Depending upon your version of Fire Brigade, you may be able to adjust the "fbdatafrom_today" macro to something like "earliest=@d-8h" to account for the time zone differences.
Thanks for your answer. Definitely sending me down the right track. I took a closer look at the data and it seems like the timezone is completely ignored but the time is retained. So if I am looking at the data on 4/20, the data will show as being from 4/19 at 12am. It almost seems you would want to modify the time macros to show more of a -1d@d. The best answer may event be to just modify the TA scheduler.
The RESTful dashboard (Matrix Overview) is immediate, and current, instead of the "snapshot" data derived from dbinspect. The latter is fairly disk-intensive at search time, so I opted to cache it by running once a day. Your points are perfectly valid, but it's a design decision (based in efficiency) that's now pretty well entrenched. I'm looking at ways to make the app more responsive (like being ready to go "day one" rather than waiting until midnight), and your feedback is helpful in knowing how people are using it.
Thanks for using it, and for your comments. Glad to hear it's working for you now.