Reporting

Orphaned Scheduled Search doesn't work

cpetterborg
SplunkTrust
SplunkTrust

The Orphaned Scheduled Search that runs on 6.4 is not working. I have looked at it down to the point that I know that this search doesn't work:

| rest splunk_server=local /services/saved/searches add_orphan_field=1

while this one does work

| rest splunk_server=local /services/saved/searches

The latter one times out without returning any results.

Can anyone tell me why, or what I can do to make this work?

1 Solution

rphillips_splk
Splunk Employee
Splunk Employee

This appears to be a known issue where a search can fail in a rest call timeout if LDAP, SAML, requests for all users take more than 60 seconds. This bug is fixed in 6.4.6 and 6.5.3. As a workaround you can try increasing the rest timeout integer value

workaround:
Increase the timeout for | REST specified here: http://docs.splunk.com/Documentation/Splunk/6.4.1/SearchReference/Rest
| rest timeout=

example:
| rest timeout=600 splunk_server=local /services/saved/searches add_orphan_field=yes

View solution in original post

rphillips_splk
Splunk Employee
Splunk Employee

This appears to be a known issue where a search can fail in a rest call timeout if LDAP, SAML, requests for all users take more than 60 seconds. This bug is fixed in 6.4.6 and 6.5.3. As a workaround you can try increasing the rest timeout integer value

workaround:
Increase the timeout for | REST specified here: http://docs.splunk.com/Documentation/Splunk/6.4.1/SearchReference/Rest
| rest timeout=

example:
| rest timeout=600 splunk_server=local /services/saved/searches add_orphan_field=yes

rphillips_splk
Splunk Employee
Splunk Employee

In some cases increasing the REST timeout will not be sufficient. This was seen in a Splunk environment on 6.5.2 (SHC) with 350 orphaned searches & an LDAP setup with 1500 users still times out.

| rest timeout=1800 splunk_server=local /servicesNS/-/-/saved/searches add_orphan_field=yes count=0 
| search orphan=1 disabled=0 is_scheduled=1 
| eval status = if(disabled = 0, "enabled", "disabled") 
| fields title eai:acl.owner eai:acl.app eai:acl.sharing orphan status is_scheduled cron_schedule next_scheduled_time next_scheduled_time actions 
| rename title AS "search name" eai:acl.owner AS owner eai:acl.app AS app eai:acl.sharing AS sharing

The dev environment with 10 orphaned searches authenticating against the same LDAP environment did not timeout and returned the orphaned searches.

This issue is documented under:
SPL-129285 The search scheduler (SavedSplunker) has scaling problems with high disabled user count and external auth systems (SAML & LDAP)

http://docs.splunk.com/Documentation/Splunk/6.5.3/ReleaseNotes/Knownissues.

0 Karma

lukasz92
Communicator

Do you have any other workaround?
How did you solve this problem?

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

We have a search head cluster with 5 SH's, and I've run it from each one of the search heads. We are all on-prem.

alt text

0 Karma

ryanoconnor
Builder

This seems network related and not as much about the add_orphan_field parameter. Are you using a load balancer for your Search Head Cluster?

0 Karma

greich
Communicator

I can confirm that this fails on 6.4.0, even when run directly on a search head part of a cluster. (i.e. bypassing the load balancer)

0 Karma

twinspop
Influencer

I've now upgraded to 6.5.0 and have the same problem. With nearly 7000 users' worth of possible churn, I get to have a gold badge on all my search heads pretty much forever. Even after finding the culprits, which in itself is not easy, resolving them is such a pain in the ass.

0 Karma

twinspop
Influencer

Same running 6.4.4. It is not network related.

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

Nice to know I'm not crazy. Now I just have to wait for 6.5.1 to see if it's fixed. I never install x.x.0 in production. 🙂

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

Yes, and it is set to load balance the rest calls, too.

0 Karma

ryanoconnor
Builder

I'm able to run both of those searches on 6.4.1. Can you let us know what error you are receiving? Are you running these from the local Splunk system?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...