topic Re: Looking for thoughts on using lookup tables when data is indexed in Splunk Search

Looking for thoughts on using lookup tables when data is indexed

pfabrizi — Fri, 15 Sep 2017 10:37:57 GMT

I know I can create lookup tables and use them during a search. We would like to apply that same process to fields as they are indexed.
so rather making field user Paul instead of Xxxad during a search we want to do this when the event is indexed.

Is this possible?
Does this impact indexing and what are the impacts on searching?

Thanks!

Re: Looking for thoughts on using lookup tables when data is indexed

gcusello — Fri, 15 Sep 2017 10:53:06 GMT

Hi pfabrizi,
if you want to denormalize logs at index time, the only way is pre-parse them before indexing by scripting, probably with the same performance impact on indexing time but a greater license consuption.
I usually use lookups to do this, also because lookups are usually quick in searches (it depends by the rows number).
Bye.
Giuseppe

Re: Looking for thoughts on using lookup tables when data is indexed

pfabrizi — Fri, 15 Sep 2017 11:17:58 GMT

is some documentation that describes this?

Re: Looking for thoughts on using lookup tables when data is indexed

pfabrizi — Fri, 15 Sep 2017 11:19:41 GMT

can lookup tables be indexed? I have a lookup table that has 4000+ rows and it takes much longer to search than my lookup tables with only a few hundred.

Re: Looking for thoughts on using lookup tables when data is indexed

gcusello — Fri, 15 Sep 2017 11:26:46 GMT

No, to pre parse you have to create script in the language you like: we used php because we have in our team an expert on this language, choose the one you know!

Anyway this is an extreme solution that we perform only when we are obliged to pre parse a log.

You could eventually think to use summary indexes, in which you can create a denormalized table and use it for your searches.
This is the best method to accelerate dashboards, the only limit I found is that you need to have a continuos data flow because you have to schedule your summarization, if you have logs that arrive in discontinous way is more difficoult to use this solution.

Bye.
Giuseppe

Re: Looking for thoughts on using lookup tables when data is indexed

gcusello — Fri, 15 Sep 2017 11:29:04 GMT

If you have many rows, lookups aren't the best solution, think to summaries.
Bye.
Giuseppe

Re: Looking for thoughts on using lookup tables when data is indexed

pfabrizi — Fri, 15 Sep 2017 11:35:57 GMT

Thank You!

I am new to SPLUNK and haven't had any training yet.

Re: Looking for thoughts on using lookup tables when data is indexed

gcusello — Fri, 15 Sep 2017 11:40:18 GMT

Training is the best way!
anyway there are interesting free courses on https://www.splunk.com/view/SP-CAAAHSM and http://docs.splunk.com/Documentation/Splunk/latest/SearchTutorial/WelcometotheSearchTutorial
Bye.
Giuseppe