Is there a way to detect subsearch limits being exceeded in scheduled searches I notice that you can get this info from REST: | rest splunk_server=local /servicesNS/$user$/$app$/search/jobs/$search_id$ | where isnotnull('messages.error') | fields id savedsearch_name, app, user, executed_at, search, messages.* And you can kinda join this to the _audit query: index=_audit action=search (has_error_warn=true OR fully_completed_search=false OR info="bad_request") | eval savedsearch_name = if(savedsearch_name="", "Ad-hoc", savedsearch_name) | eval search_id = trim(search_id, "'") | eval search = mvindex(search, 0) | map search="| rest splunk_server=local /servicesNS/$user$/$app$/search/jobs/$search_id$ | where isnotnull('messages.error') | fields id savedsearch_name, app, user, executed_at, search, messages.*" But it doesn't really work - I get lots of rest failures reported and the output is bad. You also need to run it when the search artifacts are present. Although my plan was to run this frequently and push the result to a summary index. Has anyone had better success with this? One thought would be to ingest the data that is returned by the rest call (I presume var/run/dispatch). Or might debug level logging help? ... View more

We have an on-prem Splunk deployment with 50TB/day ingest, 22PB storage, long term retention (1 year on most indexes). We use Kafka as primary ingest method. We collect cloud logs in Azure using EventHub and Kinesis in AWS. We then use Kafka MirrorMaker to bring cloud logs into on-prem Kafka. The volume ratio is currently 95% on-prem vs 5% cloud logs by volume. Inevitably this ratio will change in favor of cloud logs. And the cost to bring cloud logs on-prem will rise. For Azure alone we estimate Microsoft charge $15k/month for 5TB/day data egress. And of course the size of our cloud pipe is an issue. Therefore we are considering an architecture which will avoid backhauling cloud logs. Our stakeholders have made it clear they want a seamless experience - no logging into multiple platforms. Is your organization in a similar position to this or have you overcome this problem?        ... View more

https://docs.splunk.com/Documentation/DashApp/0.9.0/DashApp/chartsImage When you upload an image, it is stored in the KV store. Because of this, only Enterprise admins, Cloud sc_admins, and power users can upload or delete images. If you don't have the correct role assigned to upload images, you can ask someone with the correct role to add it for you Is there some permission I can assign to users to allow them to upload images without asking an admin/power user? ... View more

Admission rules are cool but it would be great to know which ones people are using. It would also be great if the platform had a set of known bad SPL which could be toggled on. Here's a couple of obvious ones we've started with: Prevent users searching all indexes: search_type=adhoc AND index=* Prevent users using all time searches: search_type=adhoc AND search_time_range=alltime   ... View more

The internal metrics logs has information on tcp in and out. This is useful to see the traffic profile between Splunk tiers. We planning our migration to the cloud and this will help with sizing the cloud pipe. Has anyone created dashboards with this information they could share with us? ... View more

We currently operate on-prem and are considering moving to Splunk Cloud. A potential blocker is the manual process required to deploy apps in Splunk Cloud. Currently we have a fully automated SDLC pipeline. We have multiple teams who make changes across multiple apps, currently with a weekly deployment cycle but we are about to move to fully automated deployments. We are informed that we would need to replace this process - where each app would need to be manually assessed and there would be up to two days delay. I'm interesting in other large customers' experience in this respect. Did you need to change your deployment mechanisms/processes when moving to the cloud? Is it cumbersome? Did you find workarounds?   ... View more

      We need to add users to our (unauthenticated) internal proxy logs. Currently the proxy logs only identity the initiator by IP address. We have DHCP and/or windows desktop logs to link the IP to a hostname. We have windows logon events which contain the hostname and user fields. Multiple users are able to log onto certain hosts and indeed might be logged on at the same time (using fast user switching). Has anyone any advice on how to solve this problem at scale (30 million events/hour) ... View more