The Orphaned Scheduled Search that runs on 6.4 is not working. I have looked at it down to the point that I know that this search doesn't work:
| rest splunk_server=local /services/saved/searches add_orphan_field=1
while this one does work
| rest splunk_server=local /services/saved/searches
The latter one times out without returning any results.
Can anyone tell me why, or what I can do to make this work?
This appears to be a known issue where a search can fail in a rest call timeout if LDAP, SAML, requests for all users take more than 60 seconds. This bug is fixed in 6.4.6 and 6.5.3. As a workaround you can try increasing the rest timeout integer value
workaround:
Increase the timeout for | REST specified here: http://docs.splunk.com/Documentation/Splunk/6.4.1/SearchReference/Rest
| rest timeout=
example:
| rest timeout=600 splunk_server=local /services/saved/searches add_orphan_field=yes
This appears to be a known issue where a search can fail in a rest call timeout if LDAP, SAML, requests for all users take more than 60 seconds. This bug is fixed in 6.4.6 and 6.5.3. As a workaround you can try increasing the rest timeout integer value
workaround:
Increase the timeout for | REST specified here: http://docs.splunk.com/Documentation/Splunk/6.4.1/SearchReference/Rest
| rest timeout=
example:
| rest timeout=600 splunk_server=local /services/saved/searches add_orphan_field=yes
In some cases increasing the REST timeout will not be sufficient. This was seen in a Splunk environment on 6.5.2 (SHC) with 350 orphaned searches & an LDAP setup with 1500 users still times out.
| rest timeout=1800 splunk_server=local /servicesNS/-/-/saved/searches add_orphan_field=yes count=0
| search orphan=1 disabled=0 is_scheduled=1
| eval status = if(disabled = 0, "enabled", "disabled")
| fields title eai:acl.owner eai:acl.app eai:acl.sharing orphan status is_scheduled cron_schedule next_scheduled_time next_scheduled_time actions
| rename title AS "search name" eai:acl.owner AS owner eai:acl.app AS app eai:acl.sharing AS sharing
The dev environment with 10 orphaned searches authenticating against the same LDAP environment did not timeout and returned the orphaned searches.
This issue is documented under:
SPL-129285 The search scheduler (SavedSplunker) has scaling problems with high disabled user count and external auth systems (SAML & LDAP)
http://docs.splunk.com/Documentation/Splunk/6.5.3/ReleaseNotes/Knownissues.
Do you have any other workaround?
How did you solve this problem?
This seems network related and not as much about the add_orphan_field parameter. Are you using a load balancer for your Search Head Cluster?
I can confirm that this fails on 6.4.0, even when run directly on a search head part of a cluster. (i.e. bypassing the load balancer)
I've now upgraded to 6.5.0 and have the same problem. With nearly 7000 users' worth of possible churn, I get to have a gold badge on all my search heads pretty much forever. Even after finding the culprits, which in itself is not easy, resolving them is such a pain in the ass.
Same running 6.4.4. It is not network related.
Nice to know I'm not crazy. Now I just have to wait for 6.5.1 to see if it's fixed. I never install x.x.0 in production. 🙂
Yes, and it is set to load balance the rest calls, too.
I'm able to run both of those searches on 6.4.1. Can you let us know what error you are receiving? Are you running these from the local Splunk system?