Scheduled reports: jobs are running fine, but the reports aren't refreshed with the results.

[jonase]

Hi,

I'm having a bit of a struggle with a few of my scheduled reports. The reports aren't being updated while the jobs are finishing and producing results.

Example scenario: my reports are scheduled to be run every n th hour with cron schedule 0 */n * * *. All the reports are starting up their respective jobs just fine and in their scheduled time. They are being finished correctly and without errors. I can even see the results of each job if I click on the search job in question. Everything is fine and dandy so far.
Problem is, the jobs do not update the report! Every time I click on the report in the app, they show me old results most of the time. Sometimes, the reports are updated correctly but most of the time I click on them to have 1-4 days old results blaring at my face with the message "The following results were generated X days ago." . I then enter the recent scheduled runs to see what's up and are presented with the most recent results.

What gives, man? Is this the first real bug I've ever encountered with Splunk? Am I missing something obvious?

What I've done, to no avail:
- Cloned the reports to see if they run correctly as new reports
- Searched for internal errors (absolutely none are found)
- Searched to see if there are any skipped searches in the logs. There are only succeeded searches there.
- Increased base searches allowed (even though there's no errors suggesting this may help in the internal logs)
- Yelled at my summer intern
- Rescheduled the reports to different times
- Allowing skew
- Adding, removing and tinkering with Schedule Windows
- Changing the owner of the reports to different users with different roles

What I'm planning to do:
- Swing a dead chicken over my head three times at midnight during a blood moon to summon the god of fire and destruction.

Some basic troubleshooting info:
Splunk 7.1.1 (recently updated from 7.0.2)
Searches are run as an administrator
Distributed environment, 1TB per day
No SHC

It's a bummer really, since this is vital to maintain certain areas of business. Anyone have any ideas?

[salzd]

Anyone has still no better option than these "wrapper-reports"?

thanks for any advice

[jonase]

If anyone have the same problem, I have an ugly workaround for the issue.

What I did was create a regular report (not scheduled) with the loadjob command to load the latest scheduled JOB of the actual scheduled report. This way, the report gets the right results, but two reports needs to be created for each single report.

It's ugly, but at least it works.

[themalkavien]

Hi Jonase,

Thx for your workaround. We have exactly the same issue with Splunk 7.1.1
As we have a lot of reports, your workaround seems to be a little too "messy" for us. Did you find any other workaround ? Did you update to a more recent version without this issue ?

Many Thanks !

[elewis1]

The parent workaround did not work for me, but I was able to load a job using REST -> Map command.

    | rest /services/search/jobs |where label="__SAVEDSEARCHTITLE__" |sort - updated |fields sid |head 1
    |map search="loadjob $sid$"