Splunk Search

Lookup table Limits

Communicator

Is there a row or column limit for a lookup table. I currently have a lookup that has 25 columns, and 350k rows, which returns no results for the output field, but, if I reduce to two columns, and run the same search, I return results.

Tags (1)
0 Karma
1 Solution

Splunk Employee
Splunk Employee

There is not supposed to be a limit.

Now, once a lookup table file reaches a certain size (by default about 10MB), we change the way that we index the lookup table for more efficient matching. So it is possible that there is a bug with how we index larger lookup tables.

Have you also tried reducing the lookup to say 10k rows, but still 25 columns?

View solution in original post

Splunk Employee
Splunk Employee

When i was trying and testing with lookup tables, i was under the impressino that something was not working either. The field extraction that was being done from the lookup tables were not happening.

However, if i gave splunk enough time to catch up and index the lookup table, then the fields would catch up.

This was not the behavior that i was seeing with small sized lookup tables, the fields were being shown immediately.

As a sidenote, my lookup table was on the order of 300MB, so i doubt there is a limit, however it might just require splunk a little time to catch up..

0 Karma

Splunk Employee
Splunk Employee

There is not supposed to be a limit.

Now, once a lookup table file reaches a certain size (by default about 10MB), we change the way that we index the lookup table for more efficient matching. So it is possible that there is a bug with how we index larger lookup tables.

Have you also tried reducing the lookup to say 10k rows, but still 25 columns?

View solution in original post

Communicator

I see no sub directories in the lookups directory, only csv files (lookup tables). Currently in $SPLUNK_HOME/etc/system/lookups

0 Karma

Splunk Employee
Splunk Employee

do you see that .index directory next to your lookup file?

0 Karma

Communicator

I increased the max_memtable_bytes=200000000, which is roughly 190MB, but still couldn't the 350K row, 25 column, 100MB lookup file to work as it should. However, I trimmed the lookup down to 10 columns, but still kept the 350K row (40MB), and it worked.

0 Karma

Splunk Employee
Splunk Employee

Yes, in limits.conf, under the [lookup] stanza, change max_memtable_bytes to a larger number.

Another thing to try is to use the original large file, and look at the directory with your lookup file. See if there is a subdirectory called .index

And see if there is any *.tsidx files in that directory. I've seen cases where the generated index files disappear for unknown reasons. You can try deleting that .index directory and running the search again and it should to re-generate an index file.

0 Karma

Communicator

I'll try reducing row count. The original file size is around 100MB, but when I reduce the lookup to two columns, the file size is around 9MB. Is there any way to increase the 10MB size, or is that hard coded?

0 Karma

Splunk Employee
Splunk Employee

It would also help to see how you defined your lookup in transforms.conf and props.conf (if automatically applied)

0 Karma

Motivator

Can you please post the searches you are using, both the one that works and the one that doesn't? And if possible please also post the first two or three rows, including the header row of the lookup table.

0 Karma