Lookups to files larger than max_memtable_bytes report file only contains a header row

[joebensimo]

With Splunk v5 and v6, I have not been able to get lookups to work with CSV files that are larger than max_memtable_bytes.

When attempting to lookup, input, or output to a lookup file that is larger than max_memtable_bytes, I get an error stating that the file is empty.

For example:
Empty csv lookup file (contains only a header) for table 'agenthash.csv': /opt/splunk/etc/apps/search/lookups/agenthash.csv

In the past, I've worked around this (as advised by support) by increasing max_memtable_bytes. However, I now have some lookups that are larger than most, and some that are at risk of growing to be larger than max_memtable_bytes.

The documentation says that Splunk will index larger files on disk, but I've not yet been able to get this to work. How can I use huge lookup files?

[haley_swarnapat]

If you are using Windows, there is a workaround (not real solution, but it should solve your problem)

From your start menu type and search for "ODBC Data Sources"
Create a System DSN
Add "Excel Files" data source
Choose your CSV file
Now the CSV file becomes accessible via ODBC Driver, voila!

Use Splunk DBLookup to fetch data from the DSN

[joebensimo]

This continues to be a problem. It appears that Splunk's functionality to index large lookup files on disk has been broken for over a year. Is this broken? Or is there something special that needs to be done to make it work?

[dshpritz]

What version of Splunk are you running?

[cramasta]

Whats larger than most? What do you have max_memtable_bytes set to?