Is the append command's default maxout=50000 also the maximum limit, or can I override this with a larger number?

[paddygriffin]

Using an append command, it seems I can successfully set the maxout to a number less than 50000, but not increase it to anything higher?
e.g. if maxout=50001 I get only 50000 results
Is this observation correct? If so is there any way to override with a larger number?

[woodcock · Accepted solution]

Correct, that is the maximum limit. But if you share your data and your goal, it is highly likely that we can come up with a solution that does not use subsearches.

[paddygriffin]

Thanks - I see the entry in limits.conf which can be increased but I'm not anxious to do that as you can imagine.
Here's a summary of the use case below. I'm exploring several different approaches - subsearches, lookups using outputlookup, intermediate .csv files using outputcsv, transactions, joins... it seems like a simple scenario and is actually quite malleable, however several of the approaches run out of steam on scalability as the number of events gets large, others on performance...

Scenario: Splunk query to determine whether a new transaction which is performed by a company in the past hour has any historical record.
A transaction is deemed to have historical record if there is a similar transaction performed by the same company in past 90 days having the **same beneficiary name OR beneficiary account number**

[woodcock]

Given your description, this should work (when run over "last hour")

... | dedup beneficiaryName beneficiaryAccountNumber | map search="search earliest=-90d@d latest=-60m beneficiaryName=$beneficiaryName$ OR beneficiaryAccountNumber=$beneficiaryAccountNumber$"

[paddygriffin]

Nice, thank you. Beautifully simple and elegant. Only concern is scalability/ performance as the map search runs once per input and that can be a very large number. Something to test 🙂

[woodcock]

It will scale without breaking (which was your problem) but it is slow and will get slower but that's a good tradeoff for not breaking down completely.

[paddygriffin]

Interestingly, there seems to be some limiter on number of results but I can't find any entry in limits.conf to explain it.
In my lab environment, it behaves fine until I exceed 60000 results, once I exceed that it returns only the first 50000 results. Up to that point performance seems linear, execution time increasing with the number of results as expected.

[woodcock]

You can add max_searches=9999 or similar to the map command, otherwise it defaults to 10, I think.

[woodcock]

OK, I have another way; try this (run for last 90 days):

... | addinfo | eval type = ((info_search_time = _time) <= 3600), "LAST_HOUR", "LAST_90_DAYS") | fillnull beneficiaryName beneficiaryAccountNumber | stats values(type) AS Types dc(type) AS NumTypes by beneficiaryName beneficiaryAccountNumber | eval RecordHistory=if((NumTypes==2), "YES", "NO")