Deployment Architecture

RAM usage at 99% on Search head

rdelmark
Explorer

Today one of my users did a search and it took 99% of the 32GB of RAM on the Splunk Production host and made it unusable for others, this must be a bug, no users could log into splunk and all alerts ceased to go out. How can one poor search kills the box. Any help would be appreciated.

This user only has 2 indexes as his default.

The user ran this search.....over a 1 day period
source="/proj/logs/prd/*systemout.log" UserName can't be null

He should have run this search........
index=claims source="/proj/logs/prd/*systemout.log" "UserName can't be null"

Tags (1)
0 Karma

yannK
Splunk Employee
Splunk Employee

How can one poor search kills the box.

There is unfortunately no protections against expensive searches.

But it's strange that a single quote causes such an issue.
Please fill a support case to report it.

0 Karma

rdelmark
Explorer

We found the issue, the default indexes included the OS index, as soon as we removed it, the search took 8 seconds and competed just fine

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Character substitutions with Regular Expressions

This challenge was first posted on Slack #puzzles channelFor BORE at .conf23, we had a puzzle question which ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

[Puzzles] Solve, Learn, Repeat: Matching cron expressions

This puzzle (first published here) is based on matching timestamps to cron expressions.All the timestamps ...