Dashboards & Visualizations

Scheduled Reports Not Running

JohnWright8
Path Finder

I have several dashboards that are based on scheduled reports (most all set to run at 3 AM daily with a two hour time window). Our server is regularly running using all memory on the physical server (i.e. it's run out of physical memory). The dashboards based on these reports don't seem to pick up the results of the scheduled reports even if I manually change the schedule to "hourly with a five minute window" and wait for the report to run.

What can I do to further troubleshoot this issue?

0 Karma

JohnWright8
Path Finder

Following up on this further... one person suggested that I need to go into the job and set the lifetime to 7 days:

From the docs: "You change the lifetime setting by selecting Job, Edit Job Settings and specifying the lifetime for the job." and "There are two lifetime settings, 10 minutes and 7 days. The default lifetime is 10 minutes. The lifetime starts from the moment the job is run. "

If you schedule a report for 2 AM and the default lifetime is 10 minutes then isn't scheduling the job pointless? Do I seriously have to edit each job again one by one and set it's lifetime to 7 days (why aren't other lifetime amounts supported)? Or is this "lifetime" not relevant to a scheduled report?

0 Karma

Jeremiah
Motivator

That isn't my understanding of how it is supposed to work. If you have a scheduled search, and the search has no actions, the search artifacts will stick around for 2x the search period. For example, for an hourly search, the artifacts are deleted after 2 hours. For a daily search, the artifacts are deleted after 2 days.

Check out this link, under the setting dispatch.ttl:
http://docs.splunk.com/Documentation/Splunk/6.3.2/Admin/Savedsearchesconf

the_wolverine
Champion

I'm not sure if the 2p configuration setting has ever worked. At least, I have not seen it ever work for me. The search takes the default setting of 24h (or whatever you have changed it to.)

0 Karma

JohnWright8
Path Finder

Thanks Jeremiah, I agree, this doesn't match my understanding and when I look at daily scheduled jobs they report as having an expiration date two days in the future.

0 Karma

chimell
Motivator

HI JohnWright8
Perhaps your schedule reports stop to trigger

go to Settings ---> Searches, reports, and alerts Open your schedule reports and verify the following options
For example the attributes of Alert that is :
Condition check for example always
Throttling Uncheck After triggering the alert, don't trigger it again for
Expiration specify How long Splunk keeps a record of each triggered alert.

And re test

0 Karma

Jeremiah
Motivator

Start by checking the status of your scheduled searches. Are they actually running successfully? You could be hitting a scheduled search or quota limit.

index=_internal sourcetype=scheduler user=your_username | timechart count by status

index=_internal sourcetype=scheduler user=your_username | chart count over savedsearch_name by status

What version of Splunk are you running? There was a bug prior to 6.1 related to a similar problem. See this previous Splunk answer.

JohnWright8
Path Finder

Thank you, this was very helpful. I can see that the jobs were "skipped". Apparently due to "maxsearches limit reached". Now I just have to figure out what to do about this. Thank you.

0 Karma

Jeremiah
Motivator

JohnWright8
Path Finder

Thank you for the link to the PDF Jeremiah. Basically it tells me that Splunk 6.3 is supposed to work like I thought (but of course it is failing).

0 Karma

JohnWright8
Path Finder

Oh, the version of Splunk is 6.3.0

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...