How to search information on usage for each search from all the different apps in our Splunk environment?

[ECovell]

Ladies and Gentlemen,

I have been tasked to write up a search that would give a total usage for each search from all the different apps. I have been digging through the master/deployment head and I have not found anything to work with. Should I be working directly with the search head where all the apps are running? If anyone could help with a good starting point I would be very grateful.

Thanks in advance,
Ernie,

[Ravan · Accepted solution]

If you are looking for search run time, disk usage, no of runs kind of stuff by each app, You can use the searches below and start exploring the data and do further customization.

| rest /servicesNS/-/-/search/jobs | eval diskUsageMB=diskUsage/1024/1024 | rename eai:acl.owner as user,eai:acl.app AS app_name |search user!="splunk-system-user"  |eval search=if((isnull(label) OR label=""),sid,label)| stats sum(diskUsageMB) as totalDiskUsage by user app_name search |sort - totalDiskUsage

index=_audit source=audittrail info=completed search_id=* user=* NOT (user="splunk-system-user" OR user="n/a" OR search_id=rt_*  OR savedsearch_name="") |rex field=_raw "^.*\__{1}(?P.*)\_\_" |stats  count AS No_Runs,avg(total_run_time) AS avg_run_time by savedsearch_name splunk_server user app_name|eval avg_run_time=round(avg_run_time/60,2)|sort - No_Runs avg_run_time

[twinspop]

Trying to improve on this excellent search. I've got duration and delay stats, and count combined into 1 chart by user, app and saved search name:

((index=_audit source=audittrail) OR index=_internal) savedsearch_name sourcetype!=splunkd_remote_searches  sourcetype!=splunkd_access NOT (splunk_archiver OR "bucket copy trigger")| rex field=_raw "\s(?:(?:search_id)|(?:sid))=[\'\"]?(?<search_id>[^ \'\",]+)"   | stats  max(dispatch_time) as dtime max(scheduled_time) as stime max(total_run_time) AS total_run_time last(app) as app by search_id savedsearch_name host user | eval delay=dtime-stime | stats avg(total_run_time) as avgDur max(total_run_time) as maxDur sum(total_run_time) as totalDur avg(delay) as avgDelay count values(host) as hosts by app savedsearch_name user | sort - count

[M2016G0216]

I'm getting the following error when I run twinspop search:

Error in 'rex' command: Encountered the following error while compiling the regex '\s(?:(?:search_id)|(?:sid))=[\'"]?(?<search_id>[^ \'",]+)': Regex: syntax error in subpattern name (missing terminator)

[M2016G0216]

And, when I run the search suggested by Ravan, I get the following error:

Error in 'rex' command: Encountered the following error while compiling the regex '^.*\__{1}(?P.*)\_\_': Regex: unrecognized character after (?P

[ECovell]

Thank you very much, the first search gave me a lot of information to work with!!

[Ravan]

You are welcome.

Also you can try the below app which gives slimier trends for monitoring Splunk users and searches ..etc

https://splunkbase.splunk.com/app/3241/

[somesoni2]

By usage do you mean license usage?
If yes then the license usage can be measured by index, host, sourcetype and source. Unless you've a way to map any of these values to an app, I don't think it's possible for measure license usage by app.

[ECovell]

No sir, I am not looking at license usage, my CSO is looking to clean up the searches and wants to know what searches are used and who is using them.