Try just | rest /services/data/indexes . ... View more

Hi, you can try one of the two ways: In the Cluster Master Spunk UI, you can navigate via Settings -> Indexer Clustering -> Peers, and look at the "Buckets" column. You will see bucket counts for each indexer. The numbers should be somewhat similar. | rest /services/data/indexes splunk_server=idx* | stats sum(currentDBSizeMB) as totalDBSize by splunk_server will also give you an idea of how balanced data is overall in terms of storage taken on peers. Jesse ... View more

Take a look at this https://answers.splunk.com/answers/121257/events-may-not-be-returned-in-sub-second-order-due-to-search-memory-limits-configured-in-limits-conf-search-max-rawsize-perchunk-error-after-upgrade-to-splunk-6-0-1.html ... View more

Hi, Rob, In the top output you showed, you can type c to show detailed command parameters. From the commands details you should be able to tell exactly what's running, e.g., ad-hoc searches, scheduled searches, with their associated search IDs. With the search IDs, you can go into $SPLUNK_HOME/var/log/splunk/dispatch directory to see what they actually are. In addition, you mentioned 36 concurrent jobs running; it sounds like your concurrency is higher than your available CPU cores. When you have more parallel jobs than available CPU cores, jobs wait, and everything slows down (including Splunk Web UI). On a positive note, you managed to max out your BOX, so all you need to do is to reduce your load to, say, 80% CPU, and you are good to go. Time to think about adding another box no matter what 🙂 ~Jesse ... View more

Thanks for reporting this, @mmoermans! max_mem_usage_mb is only used to cache events and result set for a particular search; it is not used to limit the 'splunkd' memory usage. Any result set that is larger than 200MB (default) will spill to disks -- you must be also seeing high disk IO activities and a ton of files that look like /statstmp_partition0_1555718872.35.srs.zst . There is a good chance that a big portion of that 62 minutes is spent in disk io. Your Network_Traffic must be ingesting 1) a huge volume of events and 2) events have high cardinality -- this, by nature, makes acceleration expensive (high memory and CPU cost). There are a few things you can try to speed things up: Use index scoping. By default, the data model will look at all indexes that contain a tag (e.g., pci ), if by chance, other indexes (that you are not really interested in accelerating as part of Network_Traffic) have the same tag, naturally they will be scanned for acceleration. So if you just scope the data model to the particular index, you have control. Chances are acceleration will be much faster. See screen attached showing where to define index scoping. Yes, you can try increasing max_mem_usage_mb . On a host with 64GB RAM, 1GB is just 1.5%, a good starting point to try. By spilling less temp results to disks, the memory overhead can also be reduced, possibly lower the total memory usage. Let me know how it goes. Jesse ... View more

But the two numbers should not be far apart though. % Committed Bytes In Use is virtual memory used against "Commit Limit", which could be higher than physical memory -- this is for performance reasons so clean pages don't get ejected due to committed size ~= Physical Limit. To make your life easier, if you see % Committed Bytes In Use is near 80%, then you are hitting memory limit -- only two ways to resolve this: add more memory (so Commit Limit will also increase) and run less load. Better approach is to capture both % Committed Bytes In Use Available Bytes If % Committed Bytes In Use is > 80%, OR Available Bytes is < 5%, your system is under extreme memory stress. Thanks. ... View more

Hi, @swatishs You can simply generate a dataset on your own with the Eventgen tool available in Splunk Base https://splunkbase.splunk.com/app/1924/#/details. It ships with event samples which you can modify to your environment. Additionally, to create a known distribution of your matching rates, you can create artificial fields that simply contain keywords like "every10", "every1M", etc., so you can use them in your searches to test, for example: dense search: 'search index={index_name} every10 | stats count' sparse search: 'search index={index_name} every1K | stats count' rare search: 'search index={index_name} every10M | stats count' ... View more

Check if version related issue via https://answers.splunk.com/answers/435993/universal-forwarder-using-high-cpu.html If indeed the file count is the issue, you can try increase the pipelines on the UF via https://docs.splunk.com/Documentation/Forwarder/7.2.5/Forwarder/Configureaforwardertohandlemultiplepipelinesets If none of those help, please look into the possibility to configure your PowerShell infrastructure to see if you can produce 1 file with 10K events all appended. ... View more

A few ways you can bombard Splunk with searches and measure times: You can script curl calls to the Splunk Web, using the REST APIs for searches. See API information, see here: https://docs.splunk.com/Documentation/Splunk/latest/RESTUM/RESTusing. Essentially you can dynamically create concurrent sessions of calls like stated in this answer. Use JMeter to issue the HTTP API calls. You will have better control of the concurrency (so you can adjust your load), and the performance report is provided by JMeter. And of course, because you are using Splunk, you can just send JTL files (JMeter result file) to Splunk and let it visualize everything for you. You can also create your own saved searches (to have dense, rare, sparse). These saved searches will run on cron schedules as specified by you (say, once every minute for rare, once every 5 minutes for dense, etc.), so they will automatically be trigger to run and generate results in Splunk's _internal index. You can then query that index for response times and event counts with something like 'search index=_internal source=*scheduler.log savedsearch_name=myperftest* ... View more

Not sure why link didn't show up. Here is the troubleshooting guide link: https://ta-nmon.readthedocs.io/en/latest/troubleshoot.html?highlight=fifo ... View more