How to find the exact saved search names in splunk ?

[Hemnaath]

Hi All, Can anyone guide me, on how to find the saved search name from the below saved search names.

index="_internal" source="*scheduler.log" savedsplunker | stats count BY user, savedsearch_name, host,status

Based on the search result, I found skipped status are getting generated from two splunk instance node

1) Search head cluster master
2) Deployment server

User: Admin & nobody

But unable to get the exact saved search name from the list, I could see the below name under saved search column

_ACCELERATE_C090FDA2-105E-4875-A110-3F13FF986151_SA-critical_security_controls_admin_2472f801659441b4_ACCELERATE

ACCELERATE_D4D707D0-38F3-4F47-A1AA-9DD305E110D0_DA-deployment_monitor_nobody_1a56f43bf8d5bf20_ACCELERATE

ACCELERATE_D4D707D0-38F3-4F47-A1AA-9DD305E110D0_search_nobody_365ca83246f2cca8_ACCELERATE

Note: Actually we are getting this message """The maximum number of concurrent auto-summarization searches on this instance has been reached" it is occurring due to currently running summarization searches have not completed and the scheduler cannot start the next summarization search. Due to which we could see some of the scheduled searches are skipped without running.

so we wanted to list out all auto-summarization searches from search head cluster and we may be able to remove some of that aren't needed before making a change that has the potential to greatly impact performance.

we are getting the list of accelerated saved search name as "ACCELERATE_D4D707D0-38F3-4F47-A1AA-9DD305E110D0_search_nobody_365ca83246f2cca8_ACCELERATE: so unable to find the exact name of it.

Kindly guide me how to get this fixed.

thanks in advance.

[jkat54]

These are coming from datamodel or report accelerations in the following apps:

SA-critical_security_controls
DA-deployment_monitor
search

So you can use this search to get their summarizations:

 | rest /servicesNS/nobody/APPNAMEHERE/admin/summarization/

And you can make a field called sid (using the summary.regular_id field) that matches exactly what you're seeing in your other search like this:

 | rest /servicesNS/nobody/nmon/admin/summarization/ | eval sid="_ACCELERATE_".'summary.regular_id'."_ACCELERATE_" | table sid

With a little more work you can probably join the two together into one search.

[Hemnaath]

Hi Jkat thanks for your effort on this, I had tried the above query to fetch the summarization details by executing the query for 24 hrs time frame from the search head cluster web console. But I am getting the following errors while executing the query.

| rest /servicesNS/nobody/SA-critical_security_controls/admin/summarization/

Error Details:

REST Processor: Failed to fetch REST endpoint uri=https://x.x.x.x:8089/servicesNS/nobody/SA-critical_security_controls/admin/summarization/?count=0 from server https://x.x.x.x:8089. Check that the URI path provided exists in the REST API. Learn More

[splunk01] REST Processor: Failed to fetch REST endpoint uri=https://127.0.0.1:8089/servicesNS/nobody/SA-critical_security_controls/admin/summarization/?count=0 from server https://127.0.0.1:8089. Check that the URI path provided exists in the REST API. Learn More

[splunk02] REST Processor: Failed to fetch REST endpoint uri=https://127.0.0.1:8089/servicesNS/nobody/SA-critical_security_controls/admin/summarization/?count=0 from server https://127.0.0.1:8089. Check that the URI path provided exists in the REST API. Learn More

[splunk03] REST Processor: Failed to fetch REST endpoint uri=https://127.0.0.1:8089/servicesNS/nobody/SA-critical_security_controls/admin/summarization/?count=0 from server https://127.0.0.1:8089. Check that the URI path provided exists in the REST API. Learn More

Splunk01,02 & 03 are the indexer nodes

Job :

Unexpected status for to fetch REST endpoint uri=https://x.x.x.x:8089/servicesNS/nobody/SA-critical_security_controls/admin/summarization/?count=0 from server=https://x.x.x.x:8089 - Forbidden

[splunk01] Unexpected status for to fetch REST endpoint uri=https://127.0.0.1:8089/servicesNS/nobody/SA-critical_security_controls/admin/summarization/?count=0 from server=https://127.0.0.1:8089 - Not Found

Kindly guide me on this please

[cmerriman]

try adding splunk_server="local" to your rest call.

[Hemnaath]

Hi cmerriman, thanks for you effort, could please tell me where to include this splunk_server="local" in the search query.

Kindly guide me on this please.

[cmerriman]

like so:

| rest /servicesNS/nobody/APPNAME/admin/summarization/ splunk_server="local"| eval sid="_ACCELERATE_".'summary.regular_id'."_ACCELERATE_"  | table *.name sid

you just need to add it to the end of your rest call.

[Hemnaath]

Hi Cmerriman, I am getting the below error when executing the above query.

query details :

| rest /servicesNS/nobody/SA-critical_security_controls/admin/summarization/ splunk_server="local"| eval sid="ACCELERATE".'summary.regular_id'."ACCELERATE" | table *.name sid

REST Processor: Failed to fetch REST endpoint uri=https://x.x.x.x:8089/servicesNS/nobody/SA-critical_security_controls/admin/summarization/?count=0 from server https://x.x.x.x:8089. Check that the URI path provided exists in the REST API

Job :
No matching filed exits

Unexpected status for to fetch REST endpoint uri=https://x.x.x.x:8089/servicesNS/nobody/SA-critical_security_controls/admin/summarization/?count=0 from server=https://x.x.x.x:8089- Forbidden

Kindly guide me on this

[jkat54]

Are you a member of a group or a user that has the Splunk admin role?

[Hemnaath]

Hi jkat, thanks for effort, I have assigned with the admin role. But still i could see this error when i execute the query. Kindly guide me on this.

[jkat54]

Does it work for the other app names?

DA-deployment_monitor
search

[Hemnaath]

Hi Jkat54, yes I had tried for other apps and fetch the saved search names that are configured to DA-deployment_monitor, sos, search apps. These apps are configured under deployment instances.

DA-deployment_monitor

[sourcetypes_summary_10m]
dispatch.earliest_time=-24h@h
dispatch.latest_time=now
action.email.inline = 1
action.email.reportServerEnabled = 0
alert.digest_mode = True
alert.suppress = 0
alert.track = 0
auto_summarize = 1
auto_summarize.dispatch.earliest_time = -3mon@d
search = sourcetypes_summary_10m

[forwarders_summary_10m]
dispatch.earliest_time=-24h@h
dispatch.latest_time=now
action.email.inline = 1
action.email.reportServerEnabled = 0
alert.digest_mode = True
alert.suppress = 0
alert.track = 0
auto_summarize = 1
auto_summarize.dispatch.earliest_time = -3mon@d
request.ui_dispatch_app = splunk_deployment_monitor
search = forwarders_summary_10m

similarly we have almost 10 saved searches name, so let me know how to fix the skipped search issue, what configuration change I should make to fix this issue.

thanks in advance.

[Hemnaath]

Hi Jkat54, Can you guide me on how to fix the skipped search issue for above mentioned saved search names.

thanks in advance.

[cmerriman]

or, does your user role have the dispatch_rest_to_indexer or rest_properties_get capability assigned to it?

[Hemnaath]

Hi Cmerriman, I had tried for other apps and fetch the saved search names that are configured to DA-deployment_monitor, sos, search apps. These apps are configured under deployment instances. I have checked the roles and capabilities assigned and found both "dispatch_rest_to_indexer or rest_properties_get capability" are not assigned to my role (admin). But still I could get the output.

DA-deployment_monitor

[sourcetypes_summary_10m]
dispatch.earliest_time=-24h@h
dispatch.latest_time=now
action.email.inline = 1
action.email.reportServerEnabled = 0
alert.digest_mode = True
alert.suppress = 0
alert.track = 0
auto_summarize = 1
auto_summarize.dispatch.earliest_time = -3mon@d
search = sourcetypes_summary_10m

[forwarders_summary_10m]
dispatch.earliest_time=-24h@h
dispatch.latest_time=now
action.email.inline = 1
action.email.reportServerEnabled = 0
alert.digest_mode = True
alert.suppress = 0
alert.track = 0
auto_summarize = 1
auto_summarize.dispatch.earliest_time = -3mon@d
request.ui_dispatch_app = splunk_deployment_monitor
search = forwarders_summary_10m

similarly we have almost 10 saved searches name, so let me know how to fix the skipped search issue, what configuration change I should make to fix this issue.

thanks in advance.

[Hemnaath]

Hi cmerriman, thanks for your effort on this, I have admin role assigned but both this capability are assigned to the admin role. Could please guide me on this.

thanks in advance.

[jkat54]

I cant get this to work but here's an attempt to make one search that identifies the accelerated searches:

index="_internal" source="*scheduler.log" savedsplunker savedsearch_name=_ACCELERATE*  | stats count BY user, savedsearch_name, host,status, app | rename savedsearch_name as sid | map maxsearches=50  search="| join sid [| rest /servicesNS/nobody/$app$/admin/summarization/ | eval sid="_ACCELERATE_".'summary.regular_id'."_ACCELERATE_" | table sid]"

[jkat54]

Actually, this is pretty good too:

 | rest /servicesNS/nobody/APPNAME/admin/summarization/ | eval sid="_ACCELERATE_".'summary.regular_id'."_ACCELERATE_"  | table *.name sid

[cmerriman]

this is a start. there are other fields you can use to add |search field=value to narrow results if you'd like. This will show you dashboards that are scheduled as well as reports. there is a field called is_scheduled if you want just scheduled searches.

|rest /servicesNS/-/-/saved/searches splunk_server="local"|table title