Installation

Is Splunk optimized to run fastest on a particular OS such as Windows?

dwraesner
Path Finder

Installed Splunk on AIX 6.1, and it's running old-dog slow. nmon shows splunkd processes as using more than 100% of CPU all the time. Can't click on any of the APPS without huge time delays.

Just did a regular install, and then loaded a bunch of the free apps. Now when clicking on any of the apps that's when delay issues begin so we really cannot do anything with Splunk at this point.

Labels (3)
0 Karma

dwaddle
SplunkTrust
SplunkTrust

... so ... Splunk on AIX is one of the worst performing choices you can make. Over the years, there have been compiler bugs and so forth that have led to Splunk being compiled on AIX with no compiler optimization. This has a profound impact on the performance. Splunk themselves have started the process of deprecating ( de-focusing? de-supporting? de-railing? ) the use of AIX in the indexer role. It is fully supported as a forwarder, but as an indexer the day will come that AIX is unsupported. (I wish I could find a URL to this support announcement, I no longer have the email.)

This may be strictly opinion, but your BEST bet for good performance and support is commodity x86 hardware running RHEL or CentOS. And when I say "commodity", I mean your typical 2-socket, 2U box with a bunch of local storage. Splunk does not NEED a SAN unless you want more storage per indexer than you can fit into 24 HDD slots in a 2U box. Even then, economically, you can add a lot of indexers for the price differential of a large SAN....

Lucas_K
Motivator

alt text

Lucas_K
Motivator

Have you checked to see that the default apps don't have a heap of savedsearches that are doing various things that app requires ie. updating summary/kvstore/csv lookups? Check the job inspector. A default install of splunk shouldn't be using 100% of cpu. What apps have been configured to do it a totally different matter.

Gui response delay is a combination of splunkweb being too busy/disk io and cpu contention. Any of those 3 items will cause slow web response.

0 Karma

aweitzman
Motivator

Splunk performance is mostly dependent on the resources you give it. Read the various links from here to help understand how much your Splunk installation might need: http://docs.splunk.com/Documentation/Splunk/latest/Capacity/IntroductiontocapacityplanningforSplunkE...

0 Karma

bmacias84
Champion

I very aware and Splunk does not do particularly well with virtualization especially if resources are over subscribed. with LPAR and other virtualization Splunk likes to occupy cores and does not do well when hypervisors do resource shuffling. Splunk is always working and is waiting for resources. If you have 24 physical procs 2 splunk hosts with 12 cpu Splunk host and lets say 6 hosts with 2 procs. You end up with cpu scheduling issues. Take a look at how long Splunk is waiting for access to physical resources.

0 Karma

dwraesner
Path Finder

That appears to be the case that Splunk is occupying the cores. Nmon shows the 2 CPU's as 4 cores, and splunkd and python are using the majority of all 4 cores.

0 Karma

bmacias84
Champion

You do know what splunk take look at sar, netpmon, tprof, or mpstat. Also you are only giving Splunk 4 CPUs I wouldn't use anything less than 8 CPUs. How many search jobs are running and process if you installed free apps.

0 Karma

dwraesner
Path Finder

We're not running any search jobs. That high CPU usage is just from Splunk executing. You gave me an idea, will remove all the apps except 1 since this is a new installation, and see if the performance improves.

0 Karma

bmacias84
Champion

Are you running Splunk on an LPAR with other LPARS?

0 Karma

dwraesner
Path Finder

It's a dedicated JS12 blade in an IBM 7998-60X Blade Center H running AIX 6.1. 2 PowerPC_Power6 CPU's running at 3.826 GHz. 10 GB's of the 20 GB's of memory is available. Using nmon splunkd processes are the heaviest CPU users at over 100% all the time, and I'm not doing anything with SPLUNK. That high CPU usage is just from the SPLUNK executing.

0 Karma

MuS
Legend

There are plenty of tuning possibilities on AIX - for example check this answer

http://answers.splunk.com/answers/6541/slow-search-performance-on-aix.html

I have a doc somewhere in my digital archive with some AIX config must do's ... I'll try to find it

0 Karma
Get Updates on the Splunk Community!

Exporting Splunk Apps

Join us on Monday, October 21 at 11 am PT | 2 pm ET!With the app export functionality, app developers and ...

Cisco Use Cases, ITSI Best Practices, and More New Articles from Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Build Your First SPL2 App!

Watch the recording now!.Do you want to SPL™, too? SPL2, Splunk's next-generation data search and preparation ...