Splunk search slowness and crashes

[rbathla]

We have Splunk version 6.5.2 installed back in March 2017.

We are observing a problem related to slowness listing observations below:
1. When we write a Splunk query and click on search, it just does not show even the search being started for atleast 10-15 seocnd before it starts showing search is running.
2. when we open Dashboards in the morning, they are being blank for quite some time (say 10-15 seconds) before data pops up. Then, they keep showingthe result for the day.

While debugging, we found that the splunk has been crashinmg quite a lot and there are lot of crash reports generated every month. We do not know if they are related but there seems to be problem with splunk server crashing.

Let me know if there is a way I can share the crash reprot to be looked at as well.

We are also looking at suggestion that can do perfomance tuning of Splunk from DB or from search query perspective.

[rbathla]

I also want to highlight that Crash was happening from long time but was not a big problem for us as such.

However slowness which is a big concenr at this time is noticed somewhere closer to time when we move from Equalogic SAN to SSD. We just did SSD migration 3-4 weeks back after which this slowness is being observed.

[traxxasbreaker]

Are you running on Linux infrastructure or Windows? If Linux, do you have THP disabled and the ulimits set to the recommended values? Either of those settings can cause performance problems, frequent crashes, and generally weird behavior.

When you open the dashboards first thing in the morning, is the screen completely blank for awhile, or does some of the UI load and it just takes a long time for the actual panels on the dashboard to populate?

[rbathla]

We are running it on Linux OS. I am checking with my system admin for Linux params and will tune it as per suggestion .

When we open dashboards first thing in the morning, dashboards open with blank panel frames.
After few seconds (say 15-20 sec), it starts showing blue moving line at bottom of each panel that represents query is running now. I believe problem lies in initiating the query.