topic Re: inputlookup not returning all the rows in csv file in Splunk Search

inputlookup not returning all the rows in csv file

keerthana_k — Wed, 11 Jun 2014 05:29:58 GMT

Hi,

I have a csv file with nearly 50000 rows. When I try to fetch all the rows using the inputlookup command, I am not able to retrieve all the 50000 rows. Only 42000 odd rows are returned.

Also, when I use this csv for lookup, for all the rows that are present after the 5000th row, lookup is not happening. However, if I take a particular row and place it within the 5000 rows, lookup happens succesfully.

Can anyone explain this strange behavior? Please let me know what changes I should make in conf files to enable succesful lookup.

I checked the max_memtable_bytes value in my setup and my csv file size is way below the limit.

Thanks,

Keerthana

Re: inputlookup not returning all the rows in csv file

strive — Wed, 16 Jul 2014 22:57:58 GMT

Check for unmatched/Orphan double quotes in your CSV files. That will cause problem and lookups wont be complete.

Re: inputlookup not returning all the rows in csv file

jizzmaster — Tue, 11 Aug 2015 18:16:32 GMT

Why would quotes affect this? A csv is split by linebreaks for rows, and commas for columns.

Re: inputlookup not returning all the rows in csv file

steveyz — Tue, 11 Aug 2015 18:20:28 GMT

newlines can appear in a quoted value, so it's not as simple as one logical row per line.

Re: inputlookup not returning all the rows in csv file

rashid47010 — Mon, 23 May 2016 07:50:21 GMT

what is the location of the file...(where you copy that file)
inputlookup ..............................?????

Re: inputlookup not returning all the rows in csv file

jmorais — Mon, 26 Feb 2018 17:24:33 GMT

I have the same problem.. Did you solve your case?

Re: inputlookup not returning all the rows in csv file

jmorais — Mon, 26 Feb 2018 17:26:52 GMT

I downvoted this post because that is not the problem, 50.000 rows always? a simple stats count returns 50.000, but in database the result is 206.000

Re: inputlookup not returning all the rows in csv file

jmorais — Mon, 26 Feb 2018 17:26:56 GMT

I downvoted this post because that is not the problem, 50.000 rows always? a simple stats count returns 50.000, but in database the result is 206.000

Re: inputlookup not returning all the rows in csv file

strive — Tue, 27 Feb 2018 11:14:55 GMT

Read the post carefully. They have 50000 rows but they were getting only 42000.
As per your comment then wouldn't they be getting all 50000 results.
More over the question talks about CSV. In case if the CSV has any unbalanced quotes then the lookup works till that point and fails after that.

Re: inputlookup not returning all the rows in csv file

strive — Tue, 27 Feb 2018 11:15:57 GMT

This post doesn't talk about stats command at all.

Re: inputlookup not returning all the rows in csv file

strive — Tue, 29 Sep 2020 18:12:32 GMT

If you need in stats command.. Here is the text from splunk docs

Memory and maximum results
In the limits.conf file, the maxresultrows setting in the [searchresults] stanza specifies the maximum number of results to return. The default value is 50,000. Increasing this limit can result in more memory usage.

The max_mem_usage_mb setting in the [default] stanza is used to limit how much memory the stats command uses to keep track of information. If the stats command reaches this limit, the command stops adding the requested fields to the search results. You can increase the limit, contingent on the available system memory.

If you are using Splunk Cloud and want to change either of these limits, file a Support ticket.