Splunk Search

inputlookup not returning all the rows in csv file

Communicator

Hi,

I have a csv file with nearly 50000 rows. When I try to fetch all the rows using the inputlookup command, I am not able to retrieve all the 50000 rows. Only 42000 odd rows are returned.

Also, when I use this csv for lookup, for all the rows that are present after the 5000th row, lookup is not happening. However, if I take a particular row and place it within the 5000 rows, lookup happens succesfully.

Can anyone explain this strange behavior? Please let me know what changes I should make in conf files to enable succesful lookup.

I checked the max_memtable_bytes value in my setup and my csv file size is way below the limit.

Thanks,

Keerthana

Tags (3)
0 Karma
1 Solution

Influencer

Check for unmatched/Orphan double quotes in your CSV files. That will cause problem and lookups wont be complete.

View solution in original post

Explorer

I have the same problem.. Did you solve your case?

0 Karma

Influencer

If you need in stats command.. Here is the text from splunk docs

Memory and maximum results
In the limits.conf file, the maxresultrows setting in the [searchresults] stanza specifies the maximum number of results to return. The default value is 50,000. Increasing this limit can result in more memory usage.

The maxmemusage_mb setting in the [default] stanza is used to limit how much memory the stats command uses to keep track of information. If the stats command reaches this limit, the command stops adding the requested fields to the search results. You can increase the limit, contingent on the available system memory.

If you are using Splunk Cloud and want to change either of these limits, file a Support ticket.

0 Karma

Communicator

what is the location of the file...(where you copy that file)
inputlookup ..............................?????

0 Karma

Influencer

Check for unmatched/Orphan double quotes in your CSV files. That will cause problem and lookups wont be complete.

View solution in original post

Explorer

I downvoted this post because that is not the problem, 50.000 rows always? a simple stats count returns 50.000, but in database the result is 206.000

0 Karma

Influencer

This post doesn't talk about stats command at all.

0 Karma

Explorer

I downvoted this post because that is not the problem, 50.000 rows always? a simple stats count returns 50.000, but in database the result is 206.000

0 Karma

Influencer

Read the post carefully. They have 50000 rows but they were getting only 42000.
As per your comment then wouldn't they be getting all 50000 results.
More over the question talks about CSV. In case if the CSV has any unbalanced quotes then the lookup works till that point and fails after that.

0 Karma

Path Finder

Why would quotes affect this? A csv is split by linebreaks for rows, and commas for columns.

0 Karma

Splunk Employee
Splunk Employee

newlines can appear in a quoted value, so it's not as simple as one logical row per line.

0 Karma