my splunk server has high CPU usage and I saw a bunch of splunkd process like below
search --id=adminadminsearch_search9xxxxx.yyyyy --maxbuckets=0 --ttl=600 --maxout=500000 --maxtime=8640000 --lookups=1 --reducefreq=10 --user=admin --pro --roles=admin:candelete:power:user
These searches seem to run periodically.
How could I look up scheduled/ad-hoc searches name by these search_ids, and furthermore, to retrieve the search query content?
You could use the search id to find the corresponding search folder in the $SPLUNK_HOME/var/run/splunk/dispatch. Inside the search folder, you could find the search content for that particular search.
@danielwan, you can use Splunk REST API to pull the search details along with Query, earliest and latest time and other details (you can also view search results). If you want to reuse/rerun the search you can use
| rest /services/search/jobs/admin_adminsearch_search9_xxxxx.yyyyy | table sid title eai:acl.app eai:acl.owner request.search searchEarliestTime searchLatestTime custom.dispatch.earliest_time custom.dispatch.latest_time
Please try and confirm. Following is the link to Splunk Documentation for Splunk REST API reference: http://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTsearch#search.2Fjobs.2F.7Bsearch_id.7...
There's already an accepted answer. However, if that's not working for you, and you have the sid, then you can use
index=_audit to get to the search. Note the search field has items that you probably don't want, so there's a couple rex commands to boil it down to only the SPL.
index=_audit search=* action=search search_id=*admin_adminsearch_search9_xxxxx.yyyyy* | rex "search='search (?<search>[^\e]+)" | rex mode=sed field=search "s/', autojoin=[^\e]+//g"
If this helps anyone, please upvote the answer!