A simple search(index="xx" source="/aa/bb/cc.log") made on my searchead takes 4 minutes to display 7.5 millon events for past 4 hours. This seems to be a very slow performance. My architecture contains 2 peer nodes and a master plus searchead which are dedicated machines.
More complex searches with regex takes enormous time. Where do i start troubleshooting this slowness.
Does inceasing IOPS for hot db (/var/opt/splunk/db) on my peer nodes, will have a postive effect on my perfomance or any other things to check on this.
How many diff apps do you have installed on your search head?
8 apps
iops improvement is good.. also, generally improving search speed is a complex task, requires lot of analysis...
https://conf.splunk.com/files/2017/slides/speed-up-your-searches.pdf
https://docs.splunk.com/Documentation/Splunk/8.0.6/Search/Writebettersearches
https://docs.splunk.com/Documentation/Splunk/8.0.6/Search/Quicktipsforoptimization
the summary indexing, data model acceleration ideas will improve search performance good.
(PS - i have given around 500+ karma points so far, received badge for that, if an answer helped you, a karma point would be nice!. we all should start "Learn, Give Back, Have Fun")