Installed Splunk on AIX 6.1, and it's running old-dog slow. nmon shows splunkd processes as using more than 100% of CPU all the time. Can't click on any of the APPS without huge time delays.
Just did a regular install, and then loaded a bunch of the free apps. Now when clicking on any of the apps that's when delay issues begin so we really cannot do anything with Splunk at this point.
... so ... Splunk on AIX is one of the worst performing choices you can make. Over the years, there have been compiler bugs and so forth that have led to Splunk being compiled on AIX with no compiler optimization. This has a profound impact on the performance. Splunk themselves have started the process of deprecating ( de-focusing? de-supporting? de-railing? ) the use of AIX in the indexer role. It is fully supported as a forwarder, but as an indexer the day will come that AIX is unsupported. (I wish I could find a URL to this support announcement, I no longer have the email.)
This may be strictly opinion, but your BEST bet for good performance and support is commodity x86 hardware running RHEL or CentOS. And when I say "commodity", I mean your typical 2-socket, 2U box with a bunch of local storage. Splunk does not NEED a SAN unless you want more storage per indexer than you can fit into 24 HDD slots in a 2U box. Even then, economically, you can add a lot of indexers for the price differential of a large SAN....
Have you checked to see that the default apps don't have a heap of savedsearches that are doing various things that app requires ie. updating summary/kvstore/csv lookups? Check the job inspector. A default install of splunk shouldn't be using 100% of cpu. What apps have been configured to do it a totally different matter.
Gui response delay is a combination of splunkweb being too busy/disk io and cpu contention. Any of those 3 items will cause slow web response.
Splunk performance is mostly dependent on the resources you give it. Read the various links from here to help understand how much your Splunk installation might need: http://docs.splunk.com/Documentation/Splunk/latest/Capacity/IntroductiontocapacityplanningforSplunkE...
I very aware and Splunk does not do particularly well with virtualization especially if resources are over subscribed. with LPAR and other virtualization Splunk likes to occupy cores and does not do well when hypervisors do resource shuffling. Splunk is always working and is waiting for resources. If you have 24 physical procs 2 splunk hosts with 12 cpu Splunk host and lets say 6 hosts with 2 procs. You end up with cpu scheduling issues. Take a look at how long Splunk is waiting for access to physical resources.
That appears to be the case that Splunk is occupying the cores. Nmon shows the 2 CPU's as 4 cores, and splunkd and python are using the majority of all 4 cores.
You do know what splunk take look at sar, netpmon, tprof, or mpstat. Also you are only giving Splunk 4 CPUs I wouldn't use anything less than 8 CPUs. How many search jobs are running and process if you installed free apps.
We're not running any search jobs. That high CPU usage is just from Splunk executing. You gave me an idea, will remove all the apps except 1 since this is a new installation, and see if the performance improves.
Are you running Splunk on an LPAR with other LPARS?
It's a dedicated JS12 blade in an IBM 7998-60X Blade Center H running AIX 6.1. 2 PowerPC_Power6 CPU's running at 3.826 GHz. 10 GB's of the 20 GB's of memory is available. Using nmon splunkd processes are the heaviest CPU users at over 100% all the time, and I'm not doing anything with SPLUNK. That high CPU usage is just from the SPLUNK executing.
There are plenty of tuning possibilities on AIX - for example check this answer
http://answers.splunk.com/answers/6541/slow-search-performance-on-aix.html
I have a doc somewhere in my digital archive with some AIX config must do's ... I'll try to find it