However, the search is very late, the CPU utilization of SH is 80%, the memory usage is also about 800 MB on average.
Therefore we stopped using the base search and described all the search sentences from the base search on each panel.
Then, the search time became very fast and the CPU usage rate of the SH and the memory usage rate also declined, but the CPU usage rate of the two IDXs was 80% and the memory usage rate was about 1200 MB on average.
Due to this change, I became unaware of how the distributed search works when using the dashboard base search function.
I will leave it to wiser heads to explain the splunk search architecture.
To address the underlying issue: if you gave us more information about the search itself, and about your organization's use of those panels, then we might be able to give more suggestions as to how to reduce the load on the system.
In order to have that high a load on the system, it seems like the searches must be fairly intensive and broad-based, and perhaps there are many people using the panels simultaneously for different data needs.
If so, then the directions to explore in attempting to reduce the load on the system would be the use of (1) summary indexes (2) acceleration of the data model, (3) extraction of required fields at index time rather than search time, and (4) scheduling the base search rather than searching it on load of the panels. Also, make sure that your panels are not auto-running the search repeatedly and unnecessarily.
Before attempting any of those, I would first look at optimizing the search language to make best use of the strengths of splunk. Often, a person coming into splunk from a more structured world (such as SQL) may instinctively use familiar methods that are less efficient than the splunk best practices for achieving the same goal. In fact, part of the reason that I've been investing so much time on these forums is in order to identify and flag all of those mismatches between efficient SQL query design and efficient SPL query design.
We'd be happy to help review the search, if you can post a version of it without posting any sensitive information.
◆ The values described in the ten input panels are assigned to the variables in the search sentence of each panel.
◆ The time range picker is common to all panels.
◆ Seven panels use one base search.
Base search (index = xxx field1 = $ arg1 $ field2 = $ arg2 $ ... etc)
| Stats sum (field 6) by field 7 | sort - sum (field 6)
I think that the rise in CPU and the change in search speed are caused by the following things.
◆ When using the base search function, only one base search that hands a large amount of results to post-processing moves, and each post-processing processes for one large amount of results, so the SH's CPU rises , And the search speed becomes slow.
◆ If you do not use the base search function (each panel is individually moved), since multiple base searches work, IDX 2 CPUs are climbing.
① Because the search changes according to the input panel, it is difficult to create a summary index.
Could you tell me more about the methods of ②, ③, ④?