I upgraded my single search head and indexer to version 6.3.1. I am aware of the CPU resource improvements with this release, however, I've noticed a slight decrease in performance (UI responsiveness) and increase CPU and Memory usage overall. Should I just assume this is the new norm and add resources to hopefully improve system response? I am running Splunk on virtual machines and have applied the best practices.
This message means your search processes are taking >1s to read initial configuration information from disk. What does the I/O subsystem underneath $SPLUNK_HOME/etc look like in your environment? If $SPLUNK_HOME/etc is networked storage, for example, there might be disk/network performance issues affecting search startup time.
I'm also receiving the alert below after upgrading:
Configuration initialization for Drive:\Program Files\Splunk\etc took longer than expected (1289ms) when dispatching a search (search ID: xxx__xxx__search__search5_1447719591.31937); this typically reflects underlying storage performance issues
Is this really a disk I/O issue?