topic Large Joins in Splunk Search

Large Joins

clyde772 — Thu, 02 Aug 2012 00:16:07 GMT

Hey Splunkers,

I am trying to join / lookup a large set of data to each other.

For example , transaction data to 20 million customer info.
so I have to lookup every transactions to 20 million customer info.

I have tried many ways to tackle this but haven't found the best solutions to this. lookups didn't work because the customer info records sized up to 2 GB. In order for splunk to be a big data engine, this seems to be a limitation.

I appreciate your expert advice. Cheers!

Re: Large Joins

lguinn2 — Thu, 02 Aug 2012 02:27:57 GMT

More information is needed! Can you give a (sanitized) example of your data and the search that you need to do?

Re: Large Joins

Paolo_Prigione — Thu, 02 Aug 2012 10:45:28 GMT

Just an idea: you could sort and split your lookup file in more, smaller files. Then apply multiple lookups in the search.

Otherwise, you could load your customer data to a REDIS cache, then use the REDIS lookup app to match it to transactions. The same approach might work with MySQL and the MySQL app, though more complex.

Let us know if you solved the problem!