Splunk Search

Why does search use up large amount of memory?

eng3
New Member

I'm trying to export raw linux audit logs to a file.  For example:

 

 

 

splunk.exe "sourcetype=linux:audit _time>xxxx _time<xxxxx" -output rawdata -maxout 0 > outputfile.txt

 

 

 

I'm trying to output a weeks worth but I'm not sure how many event records there are.  I tried setting maxout to 500000 and monitoring using task manager, I will see splunk grow to use 20GB of memory at its peak. I tried setting maxout to 1000000 and it used up all of my free memory.

That actual rawdata output is only a few hundred MB, why is it using up so much memory.

More importantly, is there a workaround or fix so it doesnt use up so much memory?  I could output in smaller time increments (daily for example) but I don't know if there might be a single day that happened to generate alot of events.  I suppose I could go down to hourly.  

 

Tags (1)
0 Karma

johnhuang
Motivator

I don't understand "searchtype=linux:audit", is it a typo for sourcetype? Try specifying the index in your search.

 

splunk.exe "searchtype=linux:audit _time>xxxx _time<xxxxx" -output rawdata -maxout 0 > outputfile.txt

 

0 Karma

eng3
New Member

oops, sorry, that was a typo

0 Karma
Get Updates on the Splunk Community!

Splunk Mobile: Your Brand-New Home Screen

Meet Your New Mobile Hub  Hello Splunk Community!  Staying connected to your data—no matter where you are—is ...

Introducing Value Insights (Beta): Understand the Business Impact your organization ...

Real progress on your strategic priorities starts with knowing the business outcomes your teams are delivering ...

Enterprise Security (ES) Essentials 8.3 is Now GA — Smarter Detections, Faster ...

As of today, Enterprise Security (ES) Essentials 8.3 is now generally available, helping SOC teams simplify ...