Using Index Summary Instead of a Lookup

[cabauah]

Hello,

Seeking everyone's expertise. We initially have saved searches (one running after the other) outputting to a lookup table to break processing/memory limit within splunk. It has come to a point though that the lookup has gotten big enough (> 50,000 records) and causing cluster replication issues in the environment.

One option we're looking is to use the index summary. Can a summary index be actually used as some sort of storage of a table instead of to "summarise" results over a period of time? We're only using it to get a snapshot output/table report of events but don't have a need for summary/time fields that are naturally created when saving output in the summary index

[woodcock]

Did you try KV Store? What is your status?

[woodcock]

Use KV Store instead, which is not part of the bundle and scales to large sizes very well.
https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/ConfigureKVstorelookups

[cabauah]

@woodcock - is kvstore definitely better? Can you share whitepaper detailing kvstore is better than summary index? Would kvstore have limits/performance implication as that's also one of the struggles hence we're trying to move to a different type of "storage". From my understanding kvstore can have problem with concurrent users accessing same data at the same time

[woodcock]

KV Store is a mongodb that runs on your Search Head and when the Indexers need to access it, they open a port to the Search Head and read what they need on the fly. So the data is never in the bundle. Huge Bundle sizes because of lookups is one of the main reasons KV Store was created. I have never heard of any problems with concurrent users accessing KV Store. Please post reference to such problems.

[cabauah]

thanks woodcock one more question? would kvstore be able to get snapshot of results for comparison? ie comparing results from this run to previous run?

[woodcock]

It can do everything that a file-based lookup can do and more. I do not understand your question but ... "probably?"

[somesoni2]

What version of SPlunk are you using? Have you explored KV Store for this? It is designed for large scaled lookup type data. See this link for comparison of the CSV lookup vs KV Store:
http://dev.splunk.com/view/SP-CAAAEY7#kvsvscsv

If you find it suitable (which I think it is) for your need, you can follow this link to start using KV store.
http://dev.splunk.com/view/kvstore/SP-CAAAEZQ

[cabauah]

version is 6.5 but soon be migrating to 7

[cabauah]

hello we're using kvstore in other part of the processing. Can you share whitepaper detailing kvstore is better than summary index? Would kvstore have limits/performance implication as that's also one of the struggles hence we're trying to move to a different type of "storage". From my understanding kvstore can have problem with concurrent users accessing same data at the same time?