We are using splunk cloud 8 .2.2105.2 (Build: 164754a2784c) . When we came into work this morning, we found a number of our dashboards are suddenly not rendering correctly, and we're at a loss to explain why. It would see some centering is no longer happening correctly, and font sizing has changed. You can see this through a very simple dashboard that I put together: <dashboard hideChrome="true" hideFilters="true" theme="light">
| makeresults count=1
| streamstats count
| eval msg = 5
| stats count</query>
<option name="underLabel">A field we should see</option>
</dashboard> When this is rendered, I get: You can see how the text ("1") is large, and cut off, and the label ("A field we should see") does not show. Whatever is going on has affected a number of our production dashboards; again, things were fine last night but this morning .... the rendering is bad. We have not had to mess with style sheets in the past to get the rendering correct, and don't think we should have to here ... What changed, and how can we get our dashboards rendering properly again? Help? Thx john
... View more
Okay, yes, this is solved. Using a yaml file to put the report in place, our default value of 5m for the allow_skew comes into play, regardless of what I set in my yaml file. I haven't found a way to reset that value this way; I can manually edit a report outside of our report area to set the allow_skew, though. To get things to work, I just need to be much more particular about the cron, and use a cron expression outside of what allow_skew will adjust ... and that's fine.
... View more
Well ... I take this back. When I do a manual edit of my scheduled job and set allow_skew to 0, I get what I expect. However, when our pipeline deploys a report (so I can't edit it directly) but my yaml file has allow_skew 0 .. I'm not getting the changes I expect. I feel I'm closer but ... not there yet.
... View more
Okay, running this query: index=_internal sourcetype=scheduler savedsearch_name=dip-summary-revenue-by-minute status=success I get this result: 9/1/20 9:24:24.796 AM 09-01-2020 14:24:24.796 + 0000 INFO SavedSplunker - savedsearch_id= " nobody ; dip-summary-revenue-by-minute ", search_type= " scheduled ", user= " admin ", app= " myapp ", savedsearch_name= " dip-summary-revenue-by-minute ", priority=default , status=success , digest_mode=1 , scheduled_time=1598970000 , window_time=-1 , dispatch_time=1598970264 , run_time=0.497 , result_count=5 , alert_actions= "", sid= " scheduler__admin__myapp__RMD5a5372a05036e6d57_at_1598970000_39293 ", suppressed=0 , thread_id= " AlertNotifierWorker-0 ", workload_pool= "" When I run my query to get the latest data right after running the above query: index=prod-summary-revenue source="summary_revenue_by_minute" | convert ctime(info_min_time) AS minTime | convert ctime(info_max_time) AS maxTime | convert ctime(info_search_time) AS searchTime | sort by -_time | table _time minTime maxTime searchTime _raw I get these results: 2020-09-01 09:18:00 09/01/2020 09:14:00.000 09/01/2020 09:19:00.000 09/01/2020 09:24:24.104 09/01/2020 09:18:00 -0500, search_name="dip-summary-revenue-by-minute", search_now=1598970000.000, info_min_time=1598969640.000, info_max_time=1598969940.000, info_search_time=1598970264.104, numOrdersByMinute=171, revenueByMinute="909.96" 2020-09-01 09:17:00 09/01/2020 09:14:00.000 09/01/2020 09:19:00.000 09/01/2020 09:24:24.104 09/01/2020 09:17:00 -0500, search_name="dip-summary-revenue-by-minute", search_now=1598970000.000, info_min_time=1598969640.000, info_max_time=1598969940.000, info_search_time=1598970264.104, numOrdersByMinute=167, revenueByMinute="882.49" 2020-09-01 09:16:00 09/01/2020 09:14:00.000 09/01/2020 09:19:00.000 09/01/2020 09:24:24.104 09/01/2020 09:16:00 -0500, search_name="dip-summary-revenue-by-minute", search_now=1598970000.000, info_min_time=1598969640.000, info_max_time=1598969940.000, info_search_time=1598970264.104, numOrdersByMinute=150, revenueByMinute="830.47" 2020-09-01 09:15:00 09/01/2020 09:14:00.000 09/01/2020 09:19:00.000 09/01/2020 09:24:24.104 09/01/2020 09:15:00 -0500, search_name="dip-summary-revenue-by-minute", search_now=1598970000.000, info_min_time=1598969640.000, info_max_time=1598969940.000, info_search_time=1598970264.104, numOrdersByMinute=122, revenueByMinute="606.37" Confirming that instead of the generated data being in the 1-6 minute old range, it's 6-11 minutes old (offset 5 minutes in the past from what we expected). btw, not sure it's that important to mention, but this is with splunk cloud. Dr_j
... View more
I have a report scheduled to run every 5 minutes (*/5 .....). This report gathers summary data from 6 minutes ago, to 1 minute ago, like this (I've removed the index & search criteria, etc., as they aren't germane): index=<index> <search_criteria> earliest=-6m@m latest=-1m@m .... When I run this with a "collect' with testmode=true, I get exactly what I'd expect ... If the current time is 18:00, I get data for 17:54, 17:55, 17:56, 17:57, and 17:58 The same query run as a report, though, with testmode=false, apparently the offset times are changed by 5 minutes and I get data for 17:49, 17:50, 17:51, 17:52, and 17:53 Is there something in the configuration that would have report offsets be altered by 5 minutes? Something I'm missing dealing with reports somehow? Running interactively vs running in a report is clearly offsetting things by 5 minutes. I've used this query to verify the times in my report query: index=<summary index> source="summary_revenue_by_minute" | convert ctime(info_min_time) AS minTime | convert ctime(info_max_time) AS maxTime | convert ctime(info_search_time) AS searchTime | sort by -_time | table _time minTime maxTime searchTime _raw I'm scratching my head & reading docs to no avail. Thoughts?
... View more
We're facing this same situation. A simple query basically doing just a "select 1" is taking 8 seconds (on rare occasions it will return in 4). Our more complex query will also take 8 seconds. When running the real query via MS Sql Server Studio, the query itself returns in sub-second time. The query is pretty basic --- selecting maybe a half dozen or so fields from the join of a few tables across user ids. The tables themselves are not very large --- and as evidenced by the sql server took, the query itself runs pretty quickly. I've inspected the splunk job, and dbquery is listed as taking ... you guessed it ... roughly 8 seconds. No further detail to indicate where the holdup is.
I, too, doubt this is some environmental factor. We looked into connection pooling, and it does look like a pool is configured and is being used, so it doesn't seem to be from connecting/disconnecting from the database.
Thoughts? Ideas? Currently our simple dashboard with 3 of these queries takes what seems to a user to be a very long time.
... View more