Splunk Server performance poor

[meenal901]

Hi,

I have a windows-based splunk instance. The server capacity is 4GB RAM. I am indexing around 50MB data per day.
After 1 month, the data loaded into splunk is around 4GB and now when i am loading my dashboad with 2 charts on it, it takes a lot of time.
Also sometimes the dashboard does not show the charts as well, although plain search returns the expected results.

Is there some way to check if the server is, may be, getting overloaded. What to do in such situation?

Thanks,
Meenal Luktuke

[lguinn2]

First, are you following the sizing recommendations and other capacity planning info in the Installation manual? My initial thought is that your server may be undersized - the server size recommendations are

• Intel x86 64-bit chip architecture
• 2 CPUs, 4 cores per CPU (8 cores total), at least 2.5 Ghz per core
• 8 GB RAM
• Standard 1 Gb Ethernet NIC, optional 2nd NIC for a management network
• Standard 64-bit Linux or Windows distribution

How many CPUs do you have? How full is your disk - and how fast is it? Splunk wants disks that can deliver 800 IOs per second. Is this a virtual machine or a physical machine?

Second, have you checked out the Splunk community wiki? It has a variety of troubleshooting information, including this general overview of performance troubleshooting.

I would look at the splunkd log (you can search it via index=_internal) to see if there are any errors or warnings being reported. The documentation, the wiki and this forum can help you understand any errors/warnings from splunkd. I would also look at your basic server performance indicators - what do CPU, memory, and disk IO statistics look like?

Finally, are you running scheduled searches or alerts? Real-time searches? Some apps run searches in the background, so be sure to check all the apps. What is the time range for the searches on the dashboards?

Usually, if your server is overloaded, your best option is to add another server. But not always. If your server is below the Splunk specification, you might first try to upgrade your server.

[lguinn2]

Thanks for the comment re: timechart - good point!

[Drainy]

I'll just add this as a comment as lguinn has thrown up a pretty comprehensive answer 😉 To me it sounds as if your CPU is under-spec. I am going to make an assumption that when you say charting you mean timechart, in this case Splunk needs to bucket the data into different time spans and this can be CPU heavy on an under-spec'ed CPU. In the same way this would also explain the slowdown overall you're experiencing