Splunk Enterprise Security

High CPU and memory usage with ESS (Enterprise Security Suite)

LukeMurphey
Champion

I am experiencing high CPU and memory usage with ESS. In some case, the resource usage is high enough to cause Splunk to crash. How can I fix this issue?

1 Solution

LukeMurphey
Champion

Synopsis:

Adding the notable index to the "Indexes searched by default" may cause correlation searches to enter a feedback loop that causes excessive resource usage.

Description:

This issue can be caused when the the notable index is indexes to be searched by default. In particular, this causes some of the ESS's correlation searches to trigger on their own findings since correlation searches included information about what originally triggered them in the notable index. Adding the notable index to the default searches indexes causes correlation searches to re-detect another finding based on the content of a prevous correlation search firing.

Solution:

The solution is to remove the notable index from the list of indexes to be searched by default.

Fixing via the CLI:

To fix the issue via the CLI, open or create the file $SPLUNK_HOME/etc/system/local/authorize.conf and change the "srchIndexesDefault" parameter for each role to exclude the notable index. Below is a sample config:

[role_admin]
srchIndexesDefault = main

Fixing via the GUI:

To fix the misconfiguration through the SplunkWeb interface use the steps defined below:

  1. Open the Manager (through the link in the top right of SplunkWeb)
  2. Open the configuration page for user roles by navigating to: Manager » Access controls » Roles
  3. For each role: remove "notable" from the "Indexes searched by default" option

View solution in original post

LukeMurphey
Champion

Synopsis:

Adding the notable index to the "Indexes searched by default" may cause correlation searches to enter a feedback loop that causes excessive resource usage.

Description:

This issue can be caused when the the notable index is indexes to be searched by default. In particular, this causes some of the ESS's correlation searches to trigger on their own findings since correlation searches included information about what originally triggered them in the notable index. Adding the notable index to the default searches indexes causes correlation searches to re-detect another finding based on the content of a prevous correlation search firing.

Solution:

The solution is to remove the notable index from the list of indexes to be searched by default.

Fixing via the CLI:

To fix the issue via the CLI, open or create the file $SPLUNK_HOME/etc/system/local/authorize.conf and change the "srchIndexesDefault" parameter for each role to exclude the notable index. Below is a sample config:

[role_admin]
srchIndexesDefault = main

Fixing via the GUI:

To fix the misconfiguration through the SplunkWeb interface use the steps defined below:

  1. Open the Manager (through the link in the top right of SplunkWeb)
  2. Open the configuration page for user roles by navigating to: Manager » Access controls » Roles
  3. For each role: remove "notable" from the "Indexes searched by default" option

rroberts
Splunk Employee
Splunk Employee

Solved it for me!

0 Karma
Get Updates on the Splunk Community!

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...