High CPU and memory usage with ESS (Enterprise Security Suite)

[LukeMurphey]

I am experiencing high CPU and memory usage with ESS. In some case, the resource usage is high enough to cause Splunk to crash. How can I fix this issue?

[LukeMurphey · Accepted solution]

Synopsis:

Adding the notable index to the "Indexes searched by default" may cause correlation searches to enter a feedback loop that causes excessive resource usage.

Description:

This issue can be caused when the the notable index is indexes to be searched by default. In particular, this causes some of the ESS's correlation searches to trigger on their own findings since correlation searches included information about what originally triggered them in the notable index. Adding the notable index to the default searches indexes causes correlation searches to re-detect another finding based on the content of a prevous correlation search firing.

Solution:

The solution is to remove the notable index from the list of indexes to be searched by default.

Fixing via the CLI:

To fix the issue via the CLI, open or create the file $SPLUNK_HOME/etc/system/local/authorize.conf and change the "srchIndexesDefault" parameter for each role to exclude the notable index. Below is a sample config:

[role_admin]
srchIndexesDefault = main

Fixing via the GUI:

To fix the misconfiguration through the SplunkWeb interface use the steps defined below:

1. Open the Manager (through the link in the top right of SplunkWeb)
2. Open the configuration page for user roles by navigating to: Manager » Access controls » Roles
3. For each role: remove "notable" from the "Indexes searched by default" option

[rroberts]

Solved it for me!