Determine number of searches per day (non-scheduled).

[rmorlen]

How do I determine the number of non-scheduled searches that are run per day. We are running pooled searchheads. Running Splunk 5.0.5.

[hexx]

If you are collecting process-level information for Splunk processes using the S.o.S app's ps_sos.sh scripted input, you can break down your daily search workload between scheduled and ad-hoc searches like so:

`set_sos_index` sourcetype=ps host=<indexer or search-head host>
| multikv
| `get_splunk_process_type`
| search type="searches"
| rex field=ARGS "_--user=(?<search_user>.*?)_--"
| rex field=ARGS "--id=(?<sid>.*?)_--"
| rex field=sid "remote_(?<search_head>[^_]*?)_"
| eval is_remote=if(like(sid,"%remote%"),"remote","local")
| eval is_scheduled=if(like(sid,"%scheduler_%"),"scheduled","ad-hoc")
| eval is_realtime=if(like(sid,"%rt_%"),"real-time","historical")
| eval is_subsearch=if(like(sid,"%subsearch_%"),"subsearch","generic")
| eval search_type=is_remote.", ".is_scheduled.", ".is_realtime
| timechart span=1d dc(sid) AS "Search count" by is_scheduled

Note that you'l need to run this search from within the context of the S.o.S app for the macros it uses to be available. You will also need for the ps_sos.sh scripted input to have been running for several days on the instance that you are targeting the search against.

[Glenn · Accepted solution]

Slight modification to somesoni2's answer, excluding searches which are incidental to the Splunk web interface usage (typeahead and history), and also removing the seemingly unnecessary eval of the date (timechart does this for you):

index=_audit action="search" search="*" NOT user="splunk-system-user" savedsearch_name="" NOT search="\'|history*" NOT search="\'typeahead*" | timechart count

[somesoni2]

This should give you what you need

index=_audit action="search" search="*" NOT user="splunk-system-user" | eval Date=strftime(_time,"%Y-%m-%d") | stats count by Date

[rmorlen]

This search also includes scheduled searches which is good but not what I am looking for. Based on your search I came up with this which seems more what I was looking for:

index=_audit action="search" search="*" savedsearch_name="" (user!="splunk-system-user" user!="rest*") | eval Date=strftime(_time,"%Y-%m-%d") | timechart span=1d count

[rmorlen]

I tried the search and it does work. I need to review the results because the counts seem really, really high.