Previously working scheduled reports are not working AND newly created reports are not working.
Creating a new test search works:
index=test1 | timechart count by status
The timechart is created, but putting this search into a report doesn't work:
index=test1 | timechart count by status | collect index=test2
... no scheduled search is run and no data is collected into index=test2
I've check my user/role permissions, and they seem fine .. admin access
I've checked the licensing, no limits reached
No recent search head changes
I've adjusted report/schedule times 5, 15, 1hr (cron & basic)
I've enabled/disabled summary indexing within reports
I've checked the search app for permissions.. admin read/write, global, sharing-config all_apps
I've deleted the original report and recreated it
Splunk 6.4 - enterprise
Another admin here was updating app permissions, refining global sets a few days before, but he assures this shouldn't be the issue.
If this was the problem, the only app I can see that may be the issue would be the Search app .. which i have admin access in.
Is there a global app permission that needs to be enabled/adjusted?
Any other advice on what to check or do?
Thank You
Check your _internal index for log_level=warn* or log_level=err*. I've seen everything stop in its tracks after a data model was removed via file system modification and not via GUI. Basically, i'm telling you to fix any error you can find and perhaps the most prominent error you'll find will be the culprit. In my case the missing data model was creating hundreds of errors per second, and I assume it clogged the pipe...
https://wiki.splunk.com/Community:TroubleshootingScheduledSearches
I'm currently using this link, among other links to troubleshoot ... no luck so far.
I may need to enable debugging soon
Look for stanzas that don't have the [. Or the ]. In saved searches and other conf files you have modified recently. Look at modified dates to narrow your search.
Check your _internal index for log_level=warn* or log_level=err*. I've seen everything stop in its tracks after a data model was removed via file system modification and not via GUI. Basically, i'm telling you to fix any error you can find and perhaps the most prominent error you'll find will be the culprit. In my case the missing data model was creating hundreds of errors per second, and I assume it clogged the pipe...
jkat54 - it turns out I did have log_level errors going on that were causing the problem. Your initial response worked!
Once I fixed the problem, the reporting started to flow ...
The problem was an app had macros that were only being shared within_itself but actually needed to be shared to "All Apps". The macro basically allowed the app to populated data into its own dashboard....
If a macro from a random app can stop all scheduling and reporting globally, I would think that is a bug.
I just don't understand how one random app error can clog the pipe... so absurd. I'm granting permissions to different groups of people allowing them access to building dashboards, reports, macros ... so it sounds like any of their errors can break everyone elses work... sigh.
index=_internal source=*scheduler.log*
index=_internal source=*splunkd.log* log_level=warn* OR log_level=err*
Using these searches strings helped me solve the problem
I appreciate your responses and assistance! Thanks -Sean
Wonderful! I marked my comment as an answer, can you please mark it as THE answer? Cheers, and yeah no clue how slight misconfigs can break scheduling! Just be sure to fix all Splunk errors before upgrades, etc. That's when it got me.
Do both indexes exist? You can't collect
data to an index unless it (1) exists and (2) you have permissions for the index. For the admin user, you should of course have permissions already.
@jkat54 - I've searched the index, no error or warn's ....
@iguinn Yes, both indexes exist. I manually run the collect command search and it populates the indexes.
I've reviewed permission for the index, user/role, app and they all look fine.
I've reviewed the permissions for the search& reporting app, those look good too.
I've also run a debug/refresh
I believe the server has been restarted .. but I will need to verify this tomorrow
The scheduled report just isn't kicking in/starting .. But a manual click on "run" populates the data into the index
Since the permissions were being worked on a couple days prior..., I still think its a permissions problem but so far things are checking out...
can you share savedsearches.conf?
I'm going to look into the savedsearches.conf, I can't share it ...
I have been researching posts stating that how the scheduler may be disabled and/or other apps are interfering with the scheduler from running so i will be looking into this...
As an fyi I'm tailing(RT search) using this search ... I may turn on debuging in a bit to see what is producded :
index=_internal source=*splunkd.log log_level=warn* OR log_level=err*