All Apps and Add-ons

Why are scheduled searches in dashboards causing a script to send hundreds of API requests and over load the search head?

pauliohughes
Explorer

Hi,

We’ve got a number of scheduled reports running over the previous months call data. These reports are then added to dashboards to ensure they load instantly.
However, when we view the dashboard, a script on the page is sending hundreds of API requests to the following endpoint per second. The name of the request is called a “control”

https://searchheadurl/en-gb/splunkd/__raw/services_jobs/scheduler__<username>_<Long_ID>;

This is coming from

common.js:241.

After an amount of time, this eventually grinds the browser to a halt as the requests never stop being sent/received and puts a lot of load onto the search head.

Observed behaviour

  • This behaviour does not occur when viewing the report directly
  • I was unable to replicate this using a query created outside of the plugin, saved as a scheduled report and adding it into a dashboard.
  • The API requests happen as soon as the panel is clicked using the edit > Add panel on a dashboard. This can either be in the CDR app or the search app (if the report is available to the search app)
1 Solution

sideview
SplunkTrust
SplunkTrust

Sorry I'm only seeing this now. For some reason Answers failed to send me a notification email when this question was posted, (nor is there one in spam folder). My apologies for missing it nonetheless. I'll try and check answers explicitly every day while the site's app notifications are broken.

-- Can you let me know what version of Splunk and what version of the app you're seeing this on?

-- let me know whether you're using a) "Save Report" and then manually modifying it to be scheduled, or b) "Create Alert" or c) "Create Scheduled Search". The path is very slightly different for each and this may be helpful later.

bottom line though - I can't reproduce the behavio yet, and in the scenario you describe the app's code isn't really involved at all. Which leaves the savedsearch config itself. Our app does use a supported mechanism to add a couple extra custom keys to savedsearches, and when you use the 'save' and 'create' buttons, we do write to those keys. (we use those keys to save and later set all the form fields in the reporting UI back to the way they were)

As to the error message, unfortunately Splunk's common.js file is about 5MB depending on your version, and even "line 241" is about 32KB of code, due to minification. I'd love to see a screenshot or any more details around that and other HTTP requests and console traffic immediately prior to the cascade of control requests.

also, one longshot idea -- what is the http status on those responses. My first thought was to notice your 'en-GB' locale and think 'undiscovered locale bug in splunk leading to a redirect loop'.

View solution in original post

0 Karma

sideview
SplunkTrust
SplunkTrust

Sorry I'm only seeing this now. For some reason Answers failed to send me a notification email when this question was posted, (nor is there one in spam folder). My apologies for missing it nonetheless. I'll try and check answers explicitly every day while the site's app notifications are broken.

-- Can you let me know what version of Splunk and what version of the app you're seeing this on?

-- let me know whether you're using a) "Save Report" and then manually modifying it to be scheduled, or b) "Create Alert" or c) "Create Scheduled Search". The path is very slightly different for each and this may be helpful later.

bottom line though - I can't reproduce the behavio yet, and in the scenario you describe the app's code isn't really involved at all. Which leaves the savedsearch config itself. Our app does use a supported mechanism to add a couple extra custom keys to savedsearches, and when you use the 'save' and 'create' buttons, we do write to those keys. (we use those keys to save and later set all the form fields in the reporting UI back to the way they were)

As to the error message, unfortunately Splunk's common.js file is about 5MB depending on your version, and even "line 241" is about 32KB of code, due to minification. I'd love to see a screenshot or any more details around that and other HTTP requests and console traffic immediately prior to the cascade of control requests.

also, one longshot idea -- what is the http status on those responses. My first thought was to notice your 'en-GB' locale and think 'undiscovered locale bug in splunk leading to a redirect loop'.

0 Karma

sideview
SplunkTrust
SplunkTrust

another ping - can you let me know what version of Splunk you're seeing this on. Also what is the http status of those responses? Thanks. I'm afraid I don't have anyone else reporting this and I can't reproduce.

0 Karma

pauliohughes
Explorer

Hi,

Sorry for the late reply on this.
I've been trying to replicate this, but I'm unable to.

I've followed the steps I did previously but I'm having no luck getting the same behaviour. The only way I can get the same behaviour is by adding the previous reports into a new dashboard. I must of done something very odd when creating those previous reports.

Feel free to ignore this for now. If I'm able to get this replicated successfully I'll provide some detailed information on what I did.

Thanks and Regards,
Paul

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...