slow splunk seaches

[gauravmsharma]

A simple search(index="xx" source="/aa/bb/cc.log") made on my searchead takes 4 minutes to display 7.5 millon events for past 4 hours. This seems to be a very slow performance. My architecture contains 2 peer nodes and a master plus searchead which are dedicated machines. 

More complex searches with regex takes enormous time. Where do i start troubleshooting this slowness.

Does inceasing IOPS for hot db (/var/opt/splunk/db) on my peer nodes, will have a postive effect on my perfomance or any other things to check on this.

[leonard_dupray]

How many diff apps do you have installed on your search head?

[gauravmsharma]

8 apps

[inventsekar]

Hi @gauravmsharma 

iops improvement is good.. also, generally improving search speed is a complex task, requires lot of analysis... 

https://conf.splunk.com/files/2017/slides/speed-up-your-searches.pdf

https://docs.splunk.com/Documentation/Splunk/8.0.6/Search/Writebettersearches

https://docs.splunk.com/Documentation/Splunk/8.0.6/Search/Quicktipsforoptimization

the summary indexing, data model acceleration ideas will improve search performance good. 

(PS - i have given around 500+ karma points so far, received badge for that, if an answer helped you, a karma point would be nice!. we all should start "Learn, Give Back, Have Fun")