Deployment Architecture

RAM usage at 99% on Search head

rdelmark
Explorer

Today one of my users did a search and it took 99% of the 32GB of RAM on the Splunk Production host and made it unusable for others, this must be a bug, no users could log into splunk and all alerts ceased to go out. How can one poor search kills the box. Any help would be appreciated.

This user only has 2 indexes as his default.

The user ran this search.....over a 1 day period
source="/proj/logs/prd/*systemout.log" UserName can't be null

He should have run this search........
index=claims source="/proj/logs/prd/*systemout.log" "UserName can't be null"

Tags (1)
0 Karma

yannK
Splunk Employee
Splunk Employee

How can one poor search kills the box.

There is unfortunately no protections against expensive searches.

But it's strange that a single quote causes such an issue.
Please fill a support case to report it.

0 Karma

rdelmark
Explorer

We found the issue, the default indexes included the OS index, as soon as we removed it, the search took 8 seconds and competed just fine

0 Karma
Get Updates on the Splunk Community!

Index This | What is broken 80% of the time by February?

December 2025 Edition   Hayyy Splunk Education Enthusiasts and the Eternally Curious!    We’re back with this ...

Unlock Faster Time-to-Value on Edge and Ingest Processor with New SPL2 Pipeline ...

Hello Splunk Community,   We're thrilled to share an exciting update that will help you manage your data more ...

Splunk MCP & Agentic AI: Machine Data Without Limits

Discover how the Splunk Model Context Protocol (MCP) Server can revolutionize the way your organization uses ...