Splunk Search

How to save memory and ignore table row where column value is less than x?

firstname
Explorer

I may use a search similar to this:

index=mock_index source=mock_source
| eval event = _raw
| stats count as frequency by event
| table event, frequency

which results in a table similar to the one below:

Event Frequency
2022-08-22 13:11:12 [stuff] apple.bean.34 [stuff] 2000
2022-08-22 14:18:22 [stuff] apple.bean.86 6 [stuff] 200
2022-08-22 15:17:42 [stuff] apple.bean.1 546 [stuff] 2

 

Some of the tables which I get from this search give an error that states the search_process_memory_usage_threshold has been exceeded.  If I know that I am not interested in rows where the frequency is less than 1,000, is there a way to limit the table so it only shows the rows above 1,000?  Would this also improve memory usage?

Labels (2)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

You can filter by frequency, but it will be after the stats command, which probably is where the memory is being used up.  Try eliminating unneeded fields.  That can save a lot of memory.

index=mock_index source=mock_source
| fields _raw - *
| eval event = _raw
| stats count as frequency by event
| where frequency > 1000
| table event, frequency

 

---
If this reply helps you, Karma would be appreciated.

View solution in original post

firstname
Explorer

This helped me in limiting the number of rows in the table, but it doesn't seem to impact memory usage.  To reduce memory usage, I had to use a more specific search than just:

index=mock_index source=mock_source

 Giving this an extra keyword reduced the memory usage.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

You can filter by frequency, but it will be after the stats command, which probably is where the memory is being used up.  Try eliminating unneeded fields.  That can save a lot of memory.

index=mock_index source=mock_source
| fields _raw - *
| eval event = _raw
| stats count as frequency by event
| where frequency > 1000
| table event, frequency

 

---
If this reply helps you, Karma would be appreciated.

firstname
Explorer

The line below helped in limiting the number of rows in the table.

| where frequency > 1000

However, this line below did not seem to impact memory usage.

| fields _raw - *

I chose to add an extra search parameter.  This seemed to reduce memory usage.  Instead of this:

index=mock_index source=mock_source

I did this:

index=mock_index source=mock_source mock_parameter
Tags (1)
Get Updates on the Splunk Community!

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...