When many users access the Splunk at the same time or even I test by open several dashboards at a time, the Splunk is occasionally not working and the application show “This page can’t be displayed”
As a workaround, I must restart Splunk to make it back to normal.
There are no issues on the VM server performance.
Our current system is One instance VM server located in our office building and access by Intranet
Linux server 3.12.49
Memory : 12 GB
CPUs : 12 vCPUs
Disk : 500 GB
Incoming data < 2 GB/day
Note that, it usually have message "Maximum concurrent search..", is it the reason that can make Splunk down?
Does anyone have any ideas?
surely it's a performance issue, at first you should check the disks IO that Splunk recommends must be at least 800 IOPS (better 1200): there are some open source tools like Bonnie++ to do that.
Anyway there are some dashboards in Splunk Monitoring Console that help you to understand if there are queues in indexing or in searching.
Remember that every search (if in a dashboard you have 10 panels, there are 10 running searches) takes and uses a CPU, so if you have more than 12 searches in the same time there is a search queue.
If in addition you're using many real time searches you overload your system.
I had a customer with some dashboards with 10 real time panels used at the same time by many users, solution was to add more indexers and replace real time serches with scheduled reports.