Deployment Architecture

RAM usage at 99% on Search head

rdelmark
Explorer

Today one of my users did a search and it took 99% of the 32GB of RAM on the Splunk Production host and made it unusable for others, this must be a bug, no users could log into splunk and all alerts ceased to go out. How can one poor search kills the box. Any help would be appreciated.

This user only has 2 indexes as his default.

The user ran this search.....over a 1 day period
source="/proj/logs/prd/*systemout.log" UserName can't be null

He should have run this search........
index=claims source="/proj/logs/prd/*systemout.log" "UserName can't be null"

Tags (1)
0 Karma

yannK
Splunk Employee
Splunk Employee

How can one poor search kills the box.

There is unfortunately no protections against expensive searches.

But it's strange that a single quote causes such an issue.
Please fill a support case to report it.

0 Karma

rdelmark
Explorer

We found the issue, the default indexes included the OS index, as soon as we removed it, the search took 8 seconds and competed just fine

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!