Reporting

How to troubleshoot why previously working scheduled reports and new scheduled searches are not running?

rewritex
Contributor

Previously working scheduled reports are not working AND newly created reports are not working.
Creating a new test search works:

index=test1 | timechart count by status 

The timechart is created, but putting this search into a report doesn't work:

index=test1 | timechart count by status | collect index=test2 

... no scheduled search is run and no data is collected into index=test2

I've check my user/role permissions, and they seem fine .. admin access
I've checked the licensing, no limits reached
No recent search head changes
I've adjusted report/schedule times 5, 15, 1hr (cron & basic)
I've enabled/disabled summary indexing within reports
I've checked the search app for permissions.. admin read/write, global, sharing-config all_apps
I've deleted the original report and recreated it
Splunk 6.4 - enterprise

Another admin here was updating app permissions, refining global sets a few days before, but he assures this shouldn't be the issue.
If this was the problem, the only app I can see that may be the issue would be the Search app .. which i have admin access in.

Is there a global app permission that needs to be enabled/adjusted?
Any other advice on what to check or do?

Thank You

0 Karma
1 Solution

jkat54
SplunkTrust
SplunkTrust

Check your _internal index for log_level=warn* or log_level=err*. I've seen everything stop in its tracks after a data model was removed via file system modification and not via GUI. Basically, i'm telling you to fix any error you can find and perhaps the most prominent error you'll find will be the culprit. In my case the missing data model was creating hundreds of errors per second, and I assume it clogged the pipe...

View solution in original post

rewritex
Contributor

https://wiki.splunk.com/Community:TroubleshootingScheduledSearches

I'm currently using this link, among other links to troubleshoot ... no luck so far.
I may need to enable debugging soon

0 Karma

jkat54
SplunkTrust
SplunkTrust

Look for stanzas that don't have the [. Or the ]. In saved searches and other conf files you have modified recently. Look at modified dates to narrow your search.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Check your _internal index for log_level=warn* or log_level=err*. I've seen everything stop in its tracks after a data model was removed via file system modification and not via GUI. Basically, i'm telling you to fix any error you can find and perhaps the most prominent error you'll find will be the culprit. In my case the missing data model was creating hundreds of errors per second, and I assume it clogged the pipe...

rewritex
Contributor

jkat54 - it turns out I did have log_level errors going on that were causing the problem. Your initial response worked!

Once I fixed the problem, the reporting started to flow ...
The problem was an app had macros that were only being shared within_itself but actually needed to be shared to "All Apps". The macro basically allowed the app to populated data into its own dashboard....

If a macro from a random app can stop all scheduling and reporting globally, I would think that is a bug.
I just don't understand how one random app error can clog the pipe... so absurd. I'm granting permissions to different groups of people allowing them access to building dashboards, reports, macros ... so it sounds like any of their errors can break everyone elses work... sigh.

index=_internal source=*scheduler.log*
index=_internal source=*splunkd.log* log_level=warn* OR log_level=err*

Using these searches strings helped me solve the problem
I appreciate your responses and assistance! Thanks -Sean

0 Karma

jkat54
SplunkTrust
SplunkTrust

Wonderful! I marked my comment as an answer, can you please mark it as THE answer? Cheers, and yeah no clue how slight misconfigs can break scheduling! Just be sure to fix all Splunk errors before upgrades, etc. That's when it got me.

0 Karma

lguinn2
Legend

Do both indexes exist? You can't collect data to an index unless it (1) exists and (2) you have permissions for the index. For the admin user, you should of course have permissions already.

0 Karma

rewritex
Contributor

@jkat54 - I've searched the index, no error or warn's ....

@iguinn Yes, both indexes exist. I manually run the collect command search and it populates the indexes.
I've reviewed permission for the index, user/role, app and they all look fine.
I've reviewed the permissions for the search& reporting app, those look good too.
I've also run a debug/refresh
I believe the server has been restarted .. but I will need to verify this tomorrow

The scheduled report just isn't kicking in/starting .. But a manual click on "run" populates the data into the index

Since the permissions were being worked on a couple days prior..., I still think its a permissions problem but so far things are checking out...

0 Karma

jkat54
SplunkTrust
SplunkTrust

can you share savedsearches.conf?

0 Karma

rewritex
Contributor

I'm going to look into the savedsearches.conf, I can't share it ...
I have been researching posts stating that how the scheduler may be disabled and/or other apps are interfering with the scheduler from running so i will be looking into this...

As an fyi I'm tailing(RT search) using this search ... I may turn on debuging in a bit to see what is producded :

index=_internal source=*splunkd.log log_level=warn* OR log_level=err*
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...