Splunk Enterprise Security

High CPU and memory usage with ESS (Enterprise Security Suite)

LukeMurphey
Champion

I am experiencing high CPU and memory usage with ESS. In some case, the resource usage is high enough to cause Splunk to crash. How can I fix this issue?

1 Solution

LukeMurphey
Champion

Synopsis:

Adding the notable index to the "Indexes searched by default" may cause correlation searches to enter a feedback loop that causes excessive resource usage.

Description:

This issue can be caused when the the notable index is indexes to be searched by default. In particular, this causes some of the ESS's correlation searches to trigger on their own findings since correlation searches included information about what originally triggered them in the notable index. Adding the notable index to the default searches indexes causes correlation searches to re-detect another finding based on the content of a prevous correlation search firing.

Solution:

The solution is to remove the notable index from the list of indexes to be searched by default.

Fixing via the CLI:

To fix the issue via the CLI, open or create the file $SPLUNK_HOME/etc/system/local/authorize.conf and change the "srchIndexesDefault" parameter for each role to exclude the notable index. Below is a sample config:

[role_admin]
srchIndexesDefault = main

Fixing via the GUI:

To fix the misconfiguration through the SplunkWeb interface use the steps defined below:

  1. Open the Manager (through the link in the top right of SplunkWeb)
  2. Open the configuration page for user roles by navigating to: Manager » Access controls » Roles
  3. For each role: remove "notable" from the "Indexes searched by default" option

View solution in original post

LukeMurphey
Champion

Synopsis:

Adding the notable index to the "Indexes searched by default" may cause correlation searches to enter a feedback loop that causes excessive resource usage.

Description:

This issue can be caused when the the notable index is indexes to be searched by default. In particular, this causes some of the ESS's correlation searches to trigger on their own findings since correlation searches included information about what originally triggered them in the notable index. Adding the notable index to the default searches indexes causes correlation searches to re-detect another finding based on the content of a prevous correlation search firing.

Solution:

The solution is to remove the notable index from the list of indexes to be searched by default.

Fixing via the CLI:

To fix the issue via the CLI, open or create the file $SPLUNK_HOME/etc/system/local/authorize.conf and change the "srchIndexesDefault" parameter for each role to exclude the notable index. Below is a sample config:

[role_admin]
srchIndexesDefault = main

Fixing via the GUI:

To fix the misconfiguration through the SplunkWeb interface use the steps defined below:

  1. Open the Manager (through the link in the top right of SplunkWeb)
  2. Open the configuration page for user roles by navigating to: Manager » Access controls » Roles
  3. For each role: remove "notable" from the "Indexes searched by default" option

rroberts
Splunk Employee
Splunk Employee

Solved it for me!

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...