topic Orphaned Scheduled Search doesn't work in Reporting

Orphaned Scheduled Search doesn't work

cpetterborg — Thu, 16 Jun 2016 20:38:51 GMT

The Orphaned Scheduled Search that runs on 6.4 is not working. I have looked at it down to the point that I know that this search doesn't work:

| rest splunk_server=local /services/saved/searches add_orphan_field=1

while this one does work

| rest splunk_server=local /services/saved/searches

The latter one times out without returning any results.

Can anyone tell me why, or what I can do to make this work?

Re: Orphaned Scheduled Search doesn't work

ryanoconnor — Thu, 16 Jun 2016 21:15:24 GMT

I'm able to run both of those searches on 6.4.1. Can you let us know what error you are receiving? Are you running these from the local Splunk system?

Re: Orphaned Scheduled Search doesn't work

cpetterborg — Mon, 20 Jun 2016 16:00:07 GMT

We have a search head cluster with 5 SH's, and I've run it from each one of the search heads. We are all on-prem.

Re: Orphaned Scheduled Search doesn't work

ryanoconnor — Tue, 29 Sep 2020 09:58:45 GMT

This seems network related and not as much about the add_orphan_field parameter. Are you using a load balancer for your Search Head Cluster?

Re: Orphaned Scheduled Search doesn't work

cpetterborg — Mon, 20 Jun 2016 16:12:26 GMT

Yes, and it is set to load balance the rest calls, too.

Re: Orphaned Scheduled Search doesn't work

greich — Mon, 07 Nov 2016 13:15:53 GMT

I can confirm that this fails on 6.4.0, even when run directly on a search head part of a cluster. (i.e. bypassing the load balancer)

Re: Orphaned Scheduled Search doesn't work

twinspop — Mon, 07 Nov 2016 13:33:38 GMT

Same running 6.4.4. It is not network related.

Re: Orphaned Scheduled Search doesn't work

cpetterborg — Mon, 07 Nov 2016 16:42:15 GMT

Nice to know I'm not crazy. Now I just have to wait for 6.5.1 to see if it's fixed. I never install x.x.0 in production. 🙂

Re: Orphaned Scheduled Search doesn't work

twinspop — Sun, 20 Nov 2016 00:17:40 GMT

I've now upgraded to 6.5.0 and have the same problem. With nearly 7000 users' worth of possible churn, I get to have a gold badge on all my search heads pretty much forever. Even after finding the culprits, which in itself is not easy, resolving them is such a pain in the ass.

Re: Orphaned Scheduled Search doesn't work

rphillips_splk — Tue, 29 Sep 2020 13:32:26 GMT

This appears to be a known issue where a search can fail in a rest call timeout if LDAP, SAML, requests for all users take more than 60 seconds. This bug is fixed in 6.4.6 and 6.5.3. As a workaround you can try increasing the rest timeout integer value

workaround:
Increase the timeout for | REST specified here: http://docs.splunk.com/Documentation/Splunk/6.4.1/SearchReference/Rest
| rest timeout=

example:
| rest timeout=600 splunk_server=local /services/saved/searches add_orphan_field=yes

Re: Orphaned Scheduled Search doesn't work

rphillips_splk — Fri, 21 Apr 2017 23:53:31 GMT

In some cases increasing the REST timeout will not be sufficient. This was seen in a Splunk environment on 6.5.2 (SHC) with 350 orphaned searches & an LDAP setup with 1500 users still times out.

| rest timeout=1800 splunk_server=local /servicesNS/-/-/saved/searches add_orphan_field=yes count=0 
| search orphan=1 disabled=0 is_scheduled=1 
| eval status = if(disabled = 0, "enabled", "disabled") 
| fields title eai:acl.owner eai:acl.app eai:acl.sharing orphan status is_scheduled cron_schedule next_scheduled_time next_scheduled_time actions 
| rename title AS "search name" eai:acl.owner AS owner eai:acl.app AS app eai:acl.sharing AS sharing

The dev environment with 10 orphaned searches authenticating against the same LDAP environment did not timeout and returned the orphaned searches.

This issue is documented under:
SPL-129285 The search scheduler (SavedSplunker) has scaling problems with high disabled user count and external auth systems (SAML & LDAP)

http://docs.splunk.com/Documentation/Splunk/6.5.3/ReleaseNotes/Knownissues.

Re: Orphaned Scheduled Search doesn't work

lukasz92 — Mon, 12 Jun 2017 12:50:46 GMT

Do you have any other workaround?
How did you solve this problem?