After upgrading my search head and indexer to Splunk 6.3.1, why am I seeing a decrease in performance and increase in CPU and memory usage?

[rmsit]

Hello, all.

I upgraded my single search head and indexer to version 6.3.1. I am aware of the CPU resource improvements with this release, however, I've noticed a slight decrease in performance (UI responsiveness) and increase CPU and Memory usage overall. Should I just assume this is the new norm and add resources to hopefully improve system response? I am running Splunk on virtual machines and have applied the best practices.

Thanks,
James

[risgupta_splunk]

This message means your search processes are taking >1s to read initial configuration information from disk. What does the I/O subsystem underneath $SPLUNK_HOME/etc look like in your environment? If $SPLUNK_HOME/etc is networked storage, for example, there might be disk/network performance issues affecting search startup time.

[rmsit]

I'm also receiving the alert below after upgrading:

Configuration initialization for Drive:\Program Files\Splunk\etc took longer than expected (1289ms) when dispatching a search (search ID: xxx__xxx__search__search5_1447719591.31937); this typically reflects underlying storage performance issues

Is this really a disk I/O issue?

[stevepraz]

What OS are you running on? I have seem similar issues and log messages on my Windows search head after upgrading to 6.3 and 6.3.1.

[rmsit]

I'm running on Windows Server 2008 R2 x64 Enterprise SP1.

[woodcock]

The majority of the benefits are on Indexers; did you upgrade them or just the Search Head?

[rmsit]

Yes, I upgraded the search head first, then the indexer.