Deployment Architecture

RAM usage at 99% on Search head

rdelmark
Explorer

Today one of my users did a search and it took 99% of the 32GB of RAM on the Splunk Production host and made it unusable for others, this must be a bug, no users could log into splunk and all alerts ceased to go out. How can one poor search kills the box. Any help would be appreciated.

This user only has 2 indexes as his default.

The user ran this search.....over a 1 day period
source="/proj/logs/prd/*systemout.log" UserName can't be null

He should have run this search........
index=claims source="/proj/logs/prd/*systemout.log" "UserName can't be null"

Tags (1)
0 Karma

yannK
Splunk Employee
Splunk Employee

How can one poor search kills the box.

There is unfortunately no protections against expensive searches.

But it's strange that a single quote causes such an issue.
Please fill a support case to report it.

0 Karma

rdelmark
Explorer

We found the issue, the default indexes included the OS index, as soon as we removed it, the search took 8 seconds and competed just fine

0 Karma
Get Updates on the Splunk Community!

Splunk Observability for AI

Don’t miss out on an exciting Tech Talk on Splunk Observability for AI!Discover how Splunk’s agentic AI ...

Splunk Enterprise Security 8.x: The Essential Upgrade for Threat Detection, ...

Watch On Demand the Tech Talk on November 6 at 11AM PT, and empower your SOC to reach new heights! Duration: ...

Splunk Observability as Code: From Zero to Dashboard

For the details on what Self-Service Observability and Observability as Code is, we have some awesome content ...