Large Joins

[clyde772]

Hey Splunkers,

I am trying to join / lookup a large set of data to each other.

For example , transaction data to 20 million customer info.
so I have to lookup every transactions to 20 million customer info.

I have tried many ways to tackle this but haven't found the best solutions to this. lookups didn't work because the customer info records sized up to 2 GB. In order for splunk to be a big data engine, this seems to be a limitation.

I appreciate your expert advice. Cheers!

[Paolo_Prigione]

Just an idea: you could sort and split your lookup file in more, smaller files. Then apply multiple lookups in the search.

Otherwise, you could load your customer data to a REDIS cache, then use the REDIS lookup app to match it to transactions. The same approach might work with MySQL and the MySQL app, though more complex.

Let us know if you solved the problem!

[lguinn2]

More information is needed! Can you give a (sanitized) example of your data and the search that you need to do?