If you have hardware to support allocation of more CPU resources to search execution, then it is possible to increase the max number of concurrent searches. The number of concurrent searches is based on CPU and affected by 2 settings in limits.conf. From limits.conf.spec: > [search] > max_searches_per_cpu = <int> > * the maximum number of concurrent searches per CPU. The system-wide number of searches > * is computed as max_searches_per_cpu x number_of_cpus + 2 > * Defaults to 2 > [scheduler] > max_searches_perc = <integer> > * the maximum number of searches the scheduler can run, as a percentage > * of the maximum number of concurrent searches, see [search] max_searches_per_cpu > * for how to set the system wide maximum number of searches > * Defaults to 25 For an 8 CPU box, the default number of max_searches_per_cpu is 18 (4x4cpu+2). It is likely, however, that such a server is capable of supporting more searches per CPU so it is safe to increment to max_searches_per_cpu=4 . Additionally, if you are running many scheduled searches for alerts or dashboards, you can find a more equitable division with max_searches_perc than the default 25%. These settings will allow you to maximize your hardware. If you find this is not adequate, consider adding a server. In general, the recommended approach to scaling is to add more CPUs via additional Splunk servers so the workload of search execution can be shared. ... View more

Ok, so I tried a few things, and this is what ended up working: NOT field2=* It would be more intuitive if this worked also: field2="" ... View more

When installing Splunk, the default settings may not account for usage outside the norm. Monitoring many or hundreds of active files falls in this category. There are 2 settings you can adjust in limits.conf to increase the indexing throughput when a large number of active files is involved: [inputproc] max_fd = <integer> * Maximum number of file descriptors that Splunk can use in the Select Processor. * The maximum value honored is half the current number of allowed file descriptors per process. (ulimit -n /setrlimit NOFILES) * If a value chosen is higher than the maximum allowed value, the maximum value is used instead. * Defaults to 32. time_before_close = <integer> * Modtime delta required before Splunk can close a file on EOF. * Tells the system not to close files that have been updated in past <integer> seconds. * Defaults to 5. For example, these settings can increase the number of files Splunk actively monitors while reducing the rate at which Splunk recycles file descriptors: [inputproc] max_fd = 256 time_before_close = 2 A more in-depth discussion on Splunk’s file monitoring system follows. In order to understand Splunk file monitoring it is useful to know: Each Splunk instance has a single monitoring thread One file descriptor is used to per source File descriptors are recycled once EOF is reached The default number of file descriptors used by Splunk is 32 (in limits.conf: max_fd = 32) For most Unix file systems, the max fds allocated to a single program is 1024 Splunk monitors files using a sliding window. At startup, Splunk will create the configured number of file descriptors in order to save some overhead in opening and closing fds. From this pool of fds, Splunk will begin monitoring the configured data inputs. When a fd reaches EOF, the fd is returned to the pool and immediately begins monitoring the next source in the queue. In past versions, Splunk created one thread per source. The overhead of managing the threads and context switching defeated the performance gains of monitoring files in parallel. Ultimately, Splunk is still constrained by I/O. By using a single thread, the context switching can be avoided and Splunk can better maximize the I/O throughput. The number of file descriptors and throughput is inversely proportional. The higher the number of fds, the lower the throughput per file descriptor. Therefore, increasing max_fd beyond a certain point will invoke diminishing returns. We believe this point to be about 256. Please Note: File monitoring improvements in Splunk 4.1 will deliver a significant performance increase. It is not clear if this tuning will be required in 4.1. Also Note: This tuning does not affecting indexing of gzip files. If you have many gzip files, then consider uncompressing them first to take advantage of Splunk's multi-threaded file monitoring. Splunk handles gzip files sequentially. ... View more

Hi Yancy, This is possible. Something to note about subsearches is the format of what is passed from the inner search to the outer search is important. If you are looking to pass a list of ReferenceIDs, then use the fields command at the end of your inner search. Otherwise, Splunk will by default pass the events themselves to the outer search. In pseudo search, your use case will look something like this: ... <my outer search> [search ReferenceID=abc | fields + ReferenceID] If you post some sample data, we can help you construct the specific query. ... View more

The good news is "YES, Splunk can index gzip files as is!" The bad news is, Splunk will monitor these files one at a time, instead of in parallel. Because it is not possible to predict the uncompressed size of a gzip file, Splunk processes these files in sequence for better control of disk allocation. With respect to performance, this is not ideal for handling 50k files so please consider uncompresing them before having Splunk monitor them to take advantage of Splunk's multi-threading file monitoring capabilities. ... View more