Deployment Architecture

RAM usage at 99% on Search head

rdelmark
Explorer

Today one of my users did a search and it took 99% of the 32GB of RAM on the Splunk Production host and made it unusable for others, this must be a bug, no users could log into splunk and all alerts ceased to go out. How can one poor search kills the box. Any help would be appreciated.

This user only has 2 indexes as his default.

The user ran this search.....over a 1 day period
source="/proj/logs/prd/*systemout.log" UserName can't be null

He should have run this search........
index=claims source="/proj/logs/prd/*systemout.log" "UserName can't be null"

Tags (1)
0 Karma

yannK
Splunk Employee
Splunk Employee

How can one poor search kills the box.

There is unfortunately no protections against expensive searches.

But it's strange that a single quote causes such an issue.
Please fill a support case to report it.

0 Karma

rdelmark
Explorer

We found the issue, the default indexes included the OS index, as soon as we removed it, the search took 8 seconds and competed just fine

0 Karma
Get Updates on the Splunk Community!

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...