Reporting

Orphaned Scheduled Search doesn't work

SplunkTrust
SplunkTrust

The Orphaned Scheduled Search that runs on 6.4 is not working. I have looked at it down to the point that I know that this search doesn't work:

| rest splunk_server=local /services/saved/searches add_orphan_field=1

while this one does work

| rest splunk_server=local /services/saved/searches

The latter one times out without returning any results.

Can anyone tell me why, or what I can do to make this work?

1 Solution

Splunk Employee
Splunk Employee

This appears to be a known issue where a search can fail in a rest call timeout if LDAP, SAML, requests for all users take more than 60 seconds. This bug is fixed in 6.4.6 and 6.5.3. As a workaround you can try increasing the rest timeout integer value

workaround:
Increase the timeout for | REST specified here: http://docs.splunk.com/Documentation/Splunk/6.4.1/SearchReference/Rest
| rest timeout=

example:
| rest timeout=600 splunkserver=local /services/saved/searches addorphan_field=yes

View solution in original post

Splunk Employee
Splunk Employee

This appears to be a known issue where a search can fail in a rest call timeout if LDAP, SAML, requests for all users take more than 60 seconds. This bug is fixed in 6.4.6 and 6.5.3. As a workaround you can try increasing the rest timeout integer value

workaround:
Increase the timeout for | REST specified here: http://docs.splunk.com/Documentation/Splunk/6.4.1/SearchReference/Rest
| rest timeout=

example:
| rest timeout=600 splunkserver=local /services/saved/searches addorphan_field=yes

View solution in original post

Splunk Employee
Splunk Employee

In some cases increasing the REST timeout will not be sufficient. This was seen in a Splunk environment on 6.5.2 (SHC) with 350 orphaned searches & an LDAP setup with 1500 users still times out.

| rest timeout=1800 splunk_server=local /servicesNS/-/-/saved/searches add_orphan_field=yes count=0 
| search orphan=1 disabled=0 is_scheduled=1 
| eval status = if(disabled = 0, "enabled", "disabled") 
| fields title eai:acl.owner eai:acl.app eai:acl.sharing orphan status is_scheduled cron_schedule next_scheduled_time next_scheduled_time actions 
| rename title AS "search name" eai:acl.owner AS owner eai:acl.app AS app eai:acl.sharing AS sharing

The dev environment with 10 orphaned searches authenticating against the same LDAP environment did not timeout and returned the orphaned searches.

This issue is documented under:
SPL-129285 The search scheduler (SavedSplunker) has scaling problems with high disabled user count and external auth systems (SAML & LDAP)

http://docs.splunk.com/Documentation/Splunk/6.5.3/ReleaseNotes/Knownissues.

0 Karma

Communicator

Do you have any other workaround?
How did you solve this problem?

0 Karma

SplunkTrust
SplunkTrust

We have a search head cluster with 5 SH's, and I've run it from each one of the search heads. We are all on-prem.

alt text

0 Karma

Builder

This seems network related and not as much about the addorphanfield parameter. Are you using a load balancer for your Search Head Cluster?

0 Karma

Communicator

I can confirm that this fails on 6.4.0, even when run directly on a search head part of a cluster. (i.e. bypassing the load balancer)

0 Karma

Influencer

I've now upgraded to 6.5.0 and have the same problem. With nearly 7000 users' worth of possible churn, I get to have a gold badge on all my search heads pretty much forever. Even after finding the culprits, which in itself is not easy, resolving them is such a pain in the ass.

0 Karma

Influencer

Same running 6.4.4. It is not network related.

0 Karma

SplunkTrust
SplunkTrust

Nice to know I'm not crazy. Now I just have to wait for 6.5.1 to see if it's fixed. I never install x.x.0 in production. 🙂

0 Karma

SplunkTrust
SplunkTrust

Yes, and it is set to load balance the rest calls, too.

0 Karma

Builder

I'm able to run both of those searches on 6.4.1. Can you let us know what error you are receiving? Are you running these from the local Splunk system?

0 Karma