Archive

Issues with Saved searches not running

Builder

I have multiple saved searches running at different times and strangely none of them are running in the weekend . When I checked the scheduler logs I dont see a single log for the saved searches on weekends bit there are logs before and after.

I am using the below search to see if searches are running

index=_internal sourcetype=scheduler savedsearch_name=* 

I checked the internal logs of the search head and I see many logs with error as below

03-30-2019 11:16:43.983 -0400 WARN  DispatchSearchMetadata - could not read metadata file: /opt/splunk/var/run/splunk/dispatch/rsa_scheduler__nobody_c3BsdW5rX2FwcF9kYl9jb25uZWN0__RMD54b8a4bb1225327ae_at_1528935000_10807_122777D6-EEF1-4A1A-BEB3-DAD58F68D73C/metadata.csv

When I ran the search manually I see results .Can anyone help me in this

0 Karma

Esteemed Legend

You are probably using either LDAP or SSO, in other words, non-local splunk accounts, and your Search Head has been rebooted. What happens in this case, is that splunk cannot validate the user because the user has not logged in since the Search Head reboot. For LDAP, you only need any user to login and all user's scheduled searches will run, but for SSO, you need each user to login before any of his scheduled searches will run. This is why many people deliberately orphan searches to be owned by nobody because that (non-)user always (sort of) exists enough for saved searches to run.

0 Karma

Builder

We are using SAML authentication and most of the searches are running under user nobody

0 Karma

Esteemed Legend

Is this one?

0 Karma

Builder

No.The user nobody is not using SAML. but there are multiple searches running on my name those also did not run. The strange thing is there are not logs in sourcetype=scheduler in the weekends (whole two days) .Last saturday I ran a saved search manually ,then all of sudden the other searches started running and from then own the logs in scheduler started to show

0 Karma

Esteemed Legend

What is your schedule definition? it is probably something like 0 0 * * 1-5 and you need it to be 0 0 * * *.

0 Karma

Builder

Hello @woodcock

Yes the searches are in the correct format .Strangely the scheduler does not run in weekends

0 Karma