Thanks Runals for the tips, I tried with the TERM() but it won't help much, I think it's because almost more than 95% of the syslog have the keyword and all the search is within the same index since I only have one index with all the data inside it.
Below are my result, and I find that by elimiating the search query:, I got the fastest time to complete:-
sourcetype="syslog" TERM(query:) earliest=-15m | timechart count by host 03:15
sourcetype="syslog" earliest=-15m | search query: | timechart count by host 02:29
sourcetype="syslog" earliest=-15m | timechart count by host 1:41
However, some of the events are without the query:, therefore, in reality, I can't eliminate the query: in the search string.
... View more
I ran below search over 11 millions record to plot a graph:-
sourcetype="syslog" | search query: | timechart count by host
The search above is already bounded by the earliest time and latest time of the last 15 minutes only.
The 4 x CPU consumed is 50% and iostat -xNh shows that the Read per second is low, only under 30, and Write per second is also low under 30.
However, the search is very slow and takes about 3 minutes to complete.
Is there anyway to speed up the search?
The point is that I have 3 similar searches running together on the same screen, and it results in 100% CPU consumption of the 4 x CPU, and it takes more than 3 minutes waiting time to complete the search.
... View more