Splunk Search

Why does search use up large amount of memory?

eng3
New Member

I'm trying to export raw linux audit logs to a file.  For example:

 

 

 

splunk.exe "sourcetype=linux:audit _time>xxxx _time<xxxxx" -output rawdata -maxout 0 > outputfile.txt

 

 

 

I'm trying to output a weeks worth but I'm not sure how many event records there are.  I tried setting maxout to 500000 and monitoring using task manager, I will see splunk grow to use 20GB of memory at its peak. I tried setting maxout to 1000000 and it used up all of my free memory.

That actual rawdata output is only a few hundred MB, why is it using up so much memory.

More importantly, is there a workaround or fix so it doesnt use up so much memory?  I could output in smaller time increments (daily for example) but I don't know if there might be a single day that happened to generate alot of events.  I suppose I could go down to hourly.  

 

Labels (1)
Tags (1)
0 Karma

johnhuang
Motivator

I don't understand "searchtype=linux:audit", is it a typo for sourcetype? Try specifying the index in your search.

 

splunk.exe "searchtype=linux:audit _time>xxxx _time<xxxxx" -output rawdata -maxout 0 > outputfile.txt

 

0 Karma

eng3
New Member

oops, sorry, that was a typo

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...