I have this process running on all my indexes:
[splunkd pid=7803] search --id=remote_SearchHead.local_scheduler__nobody__datasystems__RMD5e816c6f7615a1e8c_at_1470755400_14045 --maxbuckets=0 --ttl=60 --maxout=0 --maxtime=0 --lookups=1 --streaming --outCsv=true --user=splunk-system-user --pro --roles=admin:power:splunk-system-role:user
I can tell that it is coming from the search head and from the datasystems app, but I cannot find the search name or where it is scheduled to run to stop it. It is running as "nobody" or "splunk-system-user", so it is hard to ID who is running the search.
I can kill the pid but it comes right back.
It's a scheduled search so you'd find it's traces in scheduler logs
index=_internal sourcetype=scheduler sid="Copy the id field from your process description e.g. remote_SearchHead.local_scheduler__nobody__datasystems__RMD5e816c6f7615a1e8c_at_1470755400_14045"
The output will contain a field called savedsearch_id, which will include, owner;AppName;Saved search name.
I'm guessing you're killing the search before it's completed, so there is no search completion records in above query.
Try this alternative method/place. You already have the owner (nobody) and app name. This will give your saved search name.
index=_audit action=search search="*" NOT search="'typeahead*" NOT search="'|history*" search_id="Copy the id field from your process description " OR id="Copy the id field from your process description"
I did the search back 7 days and this is the result... Only one log entry
08-09-2016 11:28:19.963 -0500 INFO SavedSplunker - AlertNotifier::execute: queued sid=scheduler_nobodydatasystems_RMD5e816c6f7615a1e8c_at_1470755400_14045 for action execution
Thanks @somesoni2 this did not yield any results. Here is my search
index=_audit action=search search="*" NOT search="'typeahead*" NOT search="'|history*" search_id="*RMD5e816c6f7615a1e8c_at_1470755400_14045" OR id="*RMD5e816c6f7615a1e8c_at_1470755400_14045"
See you at dot conf
May be try like this, ensure to select proper time range which will include the search execution time
index=_audit action=search NOT (search="'typeahead*" OR search="'|history*" ) "*RMD5e816c6f7615a1e8c*"