Hist to add some more information on bootstrap. For large deployments, customer get concerned while running bootstrap when env have millions of buckets. As you know, bootstrapping would ensure that buckets which are already present on cluster would not be created again on the cluster. bootstrapping would just list all the buckets on S3 and would then create the buckets which are not present on the cluster. It is usually quick as well. Hence if the customer is only missing few buckets on the cluster, we can initiate bootstrapping and it would create these buckets. Noe for the Question : List all buckets on s3 for 7 million buckets -> is that still ok / fairly safe / quick? if we do want to discover these buckets, bootstrapping is the only option currently. it is not supported per index. the entire operation is detached from the usual operations of CM - it is safe and quick as well. ... View more

It looks like you are not comparing the same searches. The splunk-system-user account is used internally by Splunk to accomplish a lot of background work, so it is doing a lot of searches (and possibly alerts) that are more complex than most user searches. Ad-hoc searches run by users, typically using the Splunk GUI, have a higher priority than background or scheduled searches. They are also typically less complex. If this doesn't answer the question, then can you post a specific search that was run by both users, and give the execution time statistics for both searches? ... View more

@anupkpal, I have removed your email address just to mask any personal information. We try to keep sensitive and personal information off of posts. That being said, there are a few different options for monitoring system resources. One is the Monitoring Console. https://docs.splunk.com/Documentation/Splunk/7.0.1/DMC/DMCoverview The number of searches you can perform at any one time is more dependent on your hardware. One search request will use one CPU core on each indexer, basically. Here is some data on cores and searches. https://docs.splunk.com/Documentation/Splunk/7.0.1/Capacity/Accommodatemanysimultaneoussearches You can also use something like the queries in this answer to monitor searches being ran: https://answers.splunk.com/answers/118594/determine-number-of-searches-per-day-non-scheduled.html and you can use the job inspector of searches to see what parts of searches are taking the longest. ... View more

I have not tried it but isn't "remote_timeline_fetchall = false" good for reporting searches instead of search that you run on search window? ... View more