Monitoring Splunk

Issues with Saved searches not running

vrmandadi
Builder

I have multiple saved searches running at different times and strangely none of them are running in the weekend . When I checked the scheduler logs I dont see a single log for the saved searches on weekends bit there are logs before and after.

I am using the below search to see if searches are running

index=_internal sourcetype=scheduler savedsearch_name=* 

I checked the internal logs of the search head and I see many logs with error as below

03-30-2019 11:16:43.983 -0400 WARN  DispatchSearchMetadata - could not read metadata file: /opt/splunk/var/run/splunk/dispatch/rsa_scheduler__nobody_c3BsdW5rX2FwcF9kYl9jb25uZWN0__RMD54b8a4bb1225327ae_at_1528935000_10807_122777D6-EEF1-4A1A-BEB3-DAD58F68D73C/metadata.csv

When I ran the search manually I see results .Can anyone help me in this

0 Karma

woodcock
Esteemed Legend

You are probably using either LDAP or SSO, in other words, non-local splunk accounts, and your Search Head has been rebooted. What happens in this case, is that splunk cannot validate the user because the user has not logged in since the Search Head reboot. For LDAP, you only need any user to login and all user's scheduled searches will run, but for SSO, you need each user to login before any of his scheduled searches will run. This is why many people deliberately orphan searches to be owned by nobody because that (non-)user always (sort of) exists enough for saved searches to run.

0 Karma

vrmandadi
Builder

We are using SAML authentication and most of the searches are running under user nobody

0 Karma

woodcock
Esteemed Legend

Is this one?

0 Karma

vrmandadi
Builder

No.The user nobody is not using SAML. but there are multiple searches running on my name those also did not run. The strange thing is there are not logs in sourcetype=scheduler in the weekends (whole two days) .Last saturday I ran a saved search manually ,then all of sudden the other searches started running and from then own the logs in scheduler started to show

0 Karma

woodcock
Esteemed Legend

What is your schedule definition? it is probably something like 0 0 * * 1-5 and you need it to be 0 0 * * *.

0 Karma

vrmandadi
Builder

Hello @woodcock

Yes the searches are in the correct format .Strangely the scheduler does not run in weekends

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...