Splunk Search

How to save memory and ignore table row where column value is less than x?

firstname
Explorer

I may use a search similar to this:

index=mock_index source=mock_source
| eval event = _raw
| stats count as frequency by event
| table event, frequency

which results in a table similar to the one below:

Event Frequency
2022-08-22 13:11:12 [stuff] apple.bean.34 [stuff] 2000
2022-08-22 14:18:22 [stuff] apple.bean.86 6 [stuff] 200
2022-08-22 15:17:42 [stuff] apple.bean.1 546 [stuff] 2

 

Some of the tables which I get from this search give an error that states the search_process_memory_usage_threshold has been exceeded.  If I know that I am not interested in rows where the frequency is less than 1,000, is there a way to limit the table so it only shows the rows above 1,000?  Would this also improve memory usage?

Labels (2)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

You can filter by frequency, but it will be after the stats command, which probably is where the memory is being used up.  Try eliminating unneeded fields.  That can save a lot of memory.

index=mock_index source=mock_source
| fields _raw - *
| eval event = _raw
| stats count as frequency by event
| where frequency > 1000
| table event, frequency

 

---
If this reply helps you, Karma would be appreciated.

View solution in original post

firstname
Explorer

This helped me in limiting the number of rows in the table, but it doesn't seem to impact memory usage.  To reduce memory usage, I had to use a more specific search than just:

index=mock_index source=mock_source

 Giving this an extra keyword reduced the memory usage.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

You can filter by frequency, but it will be after the stats command, which probably is where the memory is being used up.  Try eliminating unneeded fields.  That can save a lot of memory.

index=mock_index source=mock_source
| fields _raw - *
| eval event = _raw
| stats count as frequency by event
| where frequency > 1000
| table event, frequency

 

---
If this reply helps you, Karma would be appreciated.

firstname
Explorer

The line below helped in limiting the number of rows in the table.

| where frequency > 1000

However, this line below did not seem to impact memory usage.

| fields _raw - *

I chose to add an extra search parameter.  This seemed to reduce memory usage.  Instead of this:

index=mock_index source=mock_source

I did this:

index=mock_index source=mock_source mock_parameter
Tags (1)
Get Updates on the Splunk Community!

Leveraging Detections from the Splunk Threat Research Team & Cisco Talos

  Now On Demand  Stay ahead of today’s evolving threats with the combined power of the Splunk Threat Research ...

New in Splunk Observability Cloud: Automated Archiving for Unused Metrics

Automated Archival is a new capability within Metrics Management; which is a robust usage & cost optimization ...

Calling All Security Pros: Ready to Race Through Boston?

Hey Splunkers, .conf25 is heading to Boston and we’re kicking things off with something bold, competitive, and ...