Splunk Search

How to find out where a search is originating from?

hartfoml
Motivator

I have this process running on all my indexes:

[splunkd pid=7803] search --id=remote_SearchHead.local_scheduler__nobody__datasystems__RMD5e816c6f7615a1e8c_at_1470755400_14045 --maxbuckets=0 --ttl=60 --maxout=0 --maxtime=0 --lookups=1 --streaming --outCsv=true --user=splunk-system-user --pro --roles=admin:power:splunk-system-role:user

I can tell that it is coming from the search head and from the datasystems app, but I cannot find the search name or where it is scheduled to run to stop it. It is running as "nobody" or "splunk-system-user", so it is hard to ID who is running the search.

I can kill the pid but it comes right back.

0 Karma

somesoni2
Revered Legend

It's a scheduled search so you'd find it's traces in scheduler logs

Try this

index=_internal sourcetype=scheduler sid="Copy the id field from your process description e.g. remote_SearchHead.local_scheduler__nobody__datasystems__RMD5e816c6f7615a1e8c_at_1470755400_14045" 

The output will contain a field called savedsearch_id, which will include, owner;AppName;Saved search name.

Update#1
I'm guessing you're killing the search before it's completed, so there is no search completion records in above query.

Try this alternative method/place. You already have the owner (nobody) and app name. This will give your saved search name.

index=_audit action=search search="*" NOT search="'typeahead*" NOT search="'|history*" 
search_id="Copy the id field from your process description " OR id="Copy the id field from your process description"
0 Karma

hartfoml
Motivator

I did the search back 7 days and this is the result... Only one log entry

08-09-2016 11:28:19.963 -0500 INFO SavedSplunker - AlertNotifier::execute: queued sid=scheduler_nobodydatasystems_RMD5e816c6f7615a1e8c_at_1470755400_14045 for action execution

0 Karma

somesoni2
Revered Legend

Try the updated answer.

0 Karma

hartfoml
Motivator

Thanks @somesoni2 this did not yield any results. Here is my search

index=_audit action=search search="*" NOT search="'typeahead*" NOT search="'|history*" 
 search_id="*RMD5e816c6f7615a1e8c_at_1470755400_14045" OR id="*RMD5e816c6f7615a1e8c_at_1470755400_14045"

See you at dot conf

0 Karma

somesoni2
Revered Legend

May be try like this, ensure to select proper time range which will include the search execution time
(1470755400)

index=_audit action=search NOT (search="'typeahead*" OR search="'|history*" )  "*RMD5e816c6f7615a1e8c*"
0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...