Splunk Search

How to find out where a search is originating from?

Motivator

I have this process running on all my indexes:

[splunkd pid=7803] search --id=remote_SearchHead.local_scheduler__nobody__datasystems__RMD5e816c6f7615a1e8c_at_1470755400_14045 --maxbuckets=0 --ttl=60 --maxout=0 --maxtime=0 --lookups=1 --streaming --outCsv=true --user=splunk-system-user --pro --roles=admin:power:splunk-system-role:user

I can tell that it is coming from the search head and from the datasystems app, but I cannot find the search name or where it is scheduled to run to stop it. It is running as "nobody" or "splunk-system-user", so it is hard to ID who is running the search.

I can kill the pid but it comes right back.

0 Karma

Revered Legend

It's a scheduled search so you'd find it's traces in scheduler logs

Try this

index=_internal sourcetype=scheduler sid="Copy the id field from your process description e.g. remote_SearchHead.local_scheduler__nobody__datasystems__RMD5e816c6f7615a1e8c_at_1470755400_14045" 

The output will contain a field called savedsearch_id, which will include, owner;AppName;Saved search name.

Update#1
I'm guessing you're killing the search before it's completed, so there is no search completion records in above query.

Try this alternative method/place. You already have the owner (nobody) and app name. This will give your saved search name.

index=_audit action=search search="*" NOT search="'typeahead*" NOT search="'|history*" 
search_id="Copy the id field from your process description " OR id="Copy the id field from your process description"
0 Karma

Motivator

I did the search back 7 days and this is the result... Only one log entry

08-09-2016 11:28:19.963 -0500 INFO SavedSplunker - AlertNotifier::execute: queued sid=scheduler_nobodydatasystems_RMD5e816c6f7615a1e8c_at_1470755400_14045 for action execution

0 Karma

Revered Legend

Try the updated answer.

0 Karma

Motivator

Thanks @somesoni2 this did not yield any results. Here is my search

index=_audit action=search search="*" NOT search="'typeahead*" NOT search="'|history*" 
 search_id="*RMD5e816c6f7615a1e8c_at_1470755400_14045" OR id="*RMD5e816c6f7615a1e8c_at_1470755400_14045"

See you at dot conf

0 Karma

Revered Legend

May be try like this, ensure to select proper time range which will include the search execution time
(1470755400)

index=_audit action=search NOT (search="'typeahead*" OR search="'|history*" )  "*RMD5e816c6f7615a1e8c*"
0 Karma