Monitoring Splunk

How to clear orphan scheduled searches that you cannot find, anywhere?

Glasses2
Communicator

Hi,

I have an annoying alert that is firing whenever 2 orphaned searches run on their cron schedule.

I have reassigned orphaned searches in that past without issue but these two searches I cannot find in the all configs to reassign.    I can find the orphaned searches with the following query 

 

 

 

 

| rest splunk_server=local /servicesNS/-/-/saved/searches add_orphan_field=yes count=0
| search orphan=1 disabled=0 is_scheduled=1
| eval status = if(disabled = 0, "enabled", "disabled")
| fields title eai:acl.owner eai:acl.app eai:acl.sharing orphan status is_scheduled cron_schedule next_scheduled_time next_scheduled_time actions
| rename title AS "search name" eai:acl.owner AS owner eai:acl.app AS app eai:acl.sharing AS sharing

 

 

 

 

 

When I go to settings > All configurations, set the search to All apps and owners, I cannot find the searches....

When I go to settings > All configs > Reassign KO > Orphaned, select to search all, (although there are loads of orphaned objects)  I cannot find these 2 searches causing the alerts.

When I look on the shc cluster nodes in the /opt/splunk/etc/apps/<app_name>, I cannot find them either.....   However the MC health check says the orphaned objects are on all 3 of the shc nodes.

I should also mention when I try to reassign other visible objects for these specific owners, it throws an error...

"Could not find object..."

Any advice greatly appreciated.

Thank you

 

 

0 Karma

Glasses2
Communicator

I don't know if this is the correct method, but it seems to have worked.

Using "find" command, I found the scheduled search under the /opt/splunk/etc/users/<user-name>/<app-name>  in savedsearches.conf.  Then I went in to each shc node and disabled it, then did a rolling restart.

 

 

Interestingly, under /opt/splunk/etc/users/<user-name>/<app-name>/metadata >in local.meta there was nothing for the owner, completely missing... but the search name was in there.   I have no idea how the shc got this way, but would really like to know, if anyone can explain.

 

Thank you

 

0 Karma
Get Updates on the Splunk Community!

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...