Getting Data In

Why am I unable to store more than 50,000 results in lookup table?

Mr_person
Explorer

See title, I'm using a scheduled query to prune a set of results from a lookup table, this lookup table has over 2m results, but after the prune, it's truncated down to 50,000. This exclusively happens when I schedule the lookup table with the "replace" option. Append works perfectly.

Pruning script:

 

 

| inputlookup my_lookup.csv
| where _time > relative_time(now(),"-6m")

 

 


Pruning schedule options: 

Mr_person_1-1680196469412.png

I've tried setting the output location to both my_lookup.csv and to other lookups. In both cases, 50,000 results seems to be the limit for the replaced lookup table.

Append schedule options:

Mr_person_0-1680196438861.png

Any help is appreciated.

Labels (2)
0 Karma
1 Solution

scelikok
SplunkTrust
SplunkTrust

Hi @Mr_person,

You are hitting the max_action_results on limits.conf which is default 50000. It is not a good idea to increase this limit to 2 million.

It seems your only option is using outputlookup in your search although it seems not safe.

limits.conf

[scheduler]
max_action_results = <integer>
* The maximum number of results to load when triggering an alert action.
* Default: 50000

 

If this reply helps you an upvote and "Accept as Solution" is appreciated.

View solution in original post

scelikok
SplunkTrust
SplunkTrust

Hi @Mr_person,

You are hitting the max_action_results on limits.conf which is default 50000. It is not a good idea to increase this limit to 2 million.

It seems your only option is using outputlookup in your search although it seems not safe.

limits.conf

[scheduler]
max_action_results = <integer>
* The maximum number of results to load when triggering an alert action.
* Default: 50000

 

If this reply helps you an upvote and "Accept as Solution" is appreciated.

Mr_person
Explorer

Also, I'm aware of the fact that I could change the report to use the outputlookup command inside of the search string, but that makes the report unsafe for others to open. For those looking for a workaround: 

| inputlookup my_lookup.csv
| where _time > relative_time(now(),"-6m")
| outputlookup my_lookup.csv append=false

 
Again this is unsafe and not ideal though. 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...