topic Re: Incomplete lookup results in Splunk Search

Incomplete lookup results

iKate — Tue, 07 Nov 2017 18:51:18 GMT

Hi everyone!

We've been randomly facing with rather annoying and critical issue while working with lookups:
sometimes only several entries get lookup fields when there should be many more of them. Rewriting lookup file helps in most of cases.

But it is not stable performance and means you cannot trust results especially when lookup is used in scheduled searches.

Can't say for sure what conditions cause such behaviour: often it happens with large csv files (over 1mln lines) that are rewritten on a daily basis and it happens in all Splunk versions.

This time I managed to save search logs to the same query when lookup worked incorrectly and when it worked ok (after its rewriting).
Comparison of these two files showed that they are mainly identical except several lines.

In "good" log file there is an entry, that misses in "bad" log:

INFO   CMBucketId   CMIndexId: New indexName=main inserted, mapping to id=1

Also in good log:

INFO  DispatchThread      SrchOptMetrics optimize_toJson=1

While in bad:

INFO  DispatchThread   SrchOptMetrics optimize_toJson=2

Excel with comparison is attached

Hope for your help, guys!

Re: Incomplete lookup results

DalJeanis — Tue, 07 Nov 2017 20:03:57 GMT

When you say "rewritten" on a daily basis, what, exactly do you mean? There is a time lag during propagation. Is it possible that the lookup file was updated during or shortly before the search that is in error?

Re: Incomplete lookup results

DalJeanis — Tue, 07 Nov 2017 20:04:53 GMT

It seems to me that there was a technique for shipping the lookup under a different name and then renaming it to put it in place "instantaneously". I'll have to review and see if I can find the description.

Re: Incomplete lookup results

iKate — Wed, 08 Nov 2017 19:39:14 GMT

I mean that the same search is executed every day and results rewrite csv file.
Resulting csv looks consistent and is searchable but somehow it isn't lookuped correctly.
Speaking about error do you mean error in search that creates this lookup? Then no errors. If you mean errors in other searches that could appear while "csv-writing-search" was running - _internal index didn't show anything critical in that period either.

Re: Incomplete lookup results

somesoni2 — Wed, 08 Nov 2017 20:50:07 GMT

The lookup file that's being updated daily, try to move it to etc/system/lookups instead of etc/apps/some_app/lookups. My bet is on knowledge bundle replication issue as it's a large lookup.

Re: Incomplete lookup results

DalJeanis — Wed, 08 Nov 2017 21:21:43 GMT

at what time does the lookup get rewritten? At what time do the searches run that are getting erroneous results?

Re: Incomplete lookup results

iKate — Fri, 10 Nov 2017 10:36:06 GMT

Lookup is rewritten early in the morning and last time we got erroneous results when used it in the evening.

Re: Incomplete lookup results

iKate — Fri, 10 Nov 2017 10:45:37 GMT

Thanks for your suggestion. But does knowlegde bundle replication applicable if we have one-machine installation?