I have multiple saved searches running at different times and strangely none of them are running in the weekend . When I checked the scheduler logs I dont see a single log for the saved searches on weekends bit there are logs before and after.
I am using the below search to see if searches are running
index=_internal sourcetype=scheduler savedsearch_name=*
I checked the internal logs of the search head and I see many logs with error as below
03-30-2019 11:16:43.983 -0400 WARN DispatchSearchMetadata - could not read metadata file: /opt/splunk/var/run/splunk/dispatch/rsa_scheduler__nobody_c3BsdW5rX2FwcF9kYl9jb25uZWN0__RMD54b8a4bb1225327ae_at_1528935000_10807_122777D6-EEF1-4A1A-BEB3-DAD58F68D73C/metadata.csv
When I ran the search manually I see results .Can anyone help me in this
You are probably using either
SSO, in other words, non-local splunk accounts, and your Search Head has been rebooted. What happens in this case, is that splunk cannot validate the user because the user has not logged in since the Search Head reboot. For
LDAP, you only need any user to login and all user's scheduled searches will run, but for
SSO, you need each user to login before any of his scheduled searches will run. This is why many people deliberately orphan searches to be owned by
nobody because that (non-)user always (sort of) exists enough for saved searches to run.
No.The user nobody is not using SAML. but there are multiple searches running on my name those also did not run. The strange thing is there are not logs in sourcetype=scheduler in the weekends (whole two days) .Last saturday I ran a saved search manually ,then all of sudden the other searches started running and from then own the logs in scheduler started to show