Map and Outputlookup: Why am I getting error "subsearch produced X results, truncating to maxout 10000", but results are not truncated?

[KarunK]

Hi All,

I am using a map command to pass some value to a search which needs to create 5 lookup files based on the input from map command.

The searches are running perfectly fine, but in the GUI, I am seeing a warning which says the results are getting truncated. But when I look at the CSV file, I am getting more than 10,000 results - to be accurate, 20,000 - 500,000 results. Also I am getting the same results when run all the five searches independently.

My question is, what is getting truncated?

| input lookup lookup.csv | table search_key CSV
| map search=" search index=abc " $search_key$  | stats latest(*) as * by host  | outputlookup lookup_$CSV$.csv"

INFO: [map]: Subsearch produced 595143 results, truncating to maxout 10000.
INFO: [map]: Subsearch produced 51000 results, truncating to maxout 10000.
INFO: [map]: Subsearch produced 22345 results, truncating to maxout 10000.
INFO: [map]: Subsearch produced 43687 results, truncating to maxout 10000.
INFO: [map]: Subsearch produced 69143 results, truncating to maxout 10000.

My understanding is If the subsearch results are getting truncated, then I shouldn't be seeing the more than 10,000 result in the outputlookup files.

Thanks in Advance

Regards

KKN

[woodcock]

The individual inner searches run to completion, which is why your outputcsv files are full/OK. However, when the data from each inner search is returned to the outer search, it is first truncated. Because you are not doing anything in the outer search, you can eliminate the error (truncate your own results first) like this:

| input lookup lookup.csv | table search_key CSV
| map search=" search index=abc " $search_key$  | stats latest(*) as * by host  | outputlookup lookup_$CSV$.csv | head 1"

[ryanoconnor]

The maxout is a setting in limits.conf

[subsearch] 
maxout = <integer>

 Maximum number of results to return from a subsearch.
 This value cannot be greater than or equal to 10500.
 Defaults to 10000.