Monitoring Splunk

How to clear orphan scheduled searches that you cannot find, anywhere?

Glasses2
Communicator

Hi,

I have an annoying alert that is firing whenever 2 orphaned searches run on their cron schedule.

I have reassigned orphaned searches in that past without issue but these two searches I cannot find in the all configs to reassign.    I can find the orphaned searches with the following query 

 

 

 

 

| rest splunk_server=local /servicesNS/-/-/saved/searches add_orphan_field=yes count=0
| search orphan=1 disabled=0 is_scheduled=1
| eval status = if(disabled = 0, "enabled", "disabled")
| fields title eai:acl.owner eai:acl.app eai:acl.sharing orphan status is_scheduled cron_schedule next_scheduled_time next_scheduled_time actions
| rename title AS "search name" eai:acl.owner AS owner eai:acl.app AS app eai:acl.sharing AS sharing

 

 

 

 

 

When I go to settings > All configurations, set the search to All apps and owners, I cannot find the searches....

When I go to settings > All configs > Reassign KO > Orphaned, select to search all, (although there are loads of orphaned objects)  I cannot find these 2 searches causing the alerts.

When I look on the shc cluster nodes in the /opt/splunk/etc/apps/<app_name>, I cannot find them either.....   However the MC health check says the orphaned objects are on all 3 of the shc nodes.

I should also mention when I try to reassign other visible objects for these specific owners, it throws an error...

"Could not find object..."

Any advice greatly appreciated.

Thank you

 

 

0 Karma

Glasses2
Communicator

I don't know if this is the correct method, but it seems to have worked.

Using "find" command, I found the scheduled search under the /opt/splunk/etc/users/<user-name>/<app-name>  in savedsearches.conf.  Then I went in to each shc node and disabled it, then did a rolling restart.

 

 

Interestingly, under /opt/splunk/etc/users/<user-name>/<app-name>/metadata >in local.meta there was nothing for the owner, completely missing... but the search name was in there.   I have no idea how the shc got this way, but would really like to know, if anyone can explain.

 

Thank you

 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...