topic Re: How to troubleshoot why our lookup that is updated by "outputlookup append=true" missing data added from the last 2 months? in Splunk Search

How to troubleshoot why our lookup that is updated by "outputlookup append=true" missing data added from the last 2 months?

Chamrong — Fri, 09 Jun 2017 20:34:14 GMT

We have small lookup updated in search by outputlookup append=true
This is a SMALL size
Our users noticed the lookup lost the added data from the last 2 months.
Any clue?
We have a search head and indexer cluster.

Re: How to troubleshoot why our lookup that is updated by "outputlookup append=true" missing data added from the last 2 months?

mattymo — Fri, 09 Jun 2017 21:07:16 GMT

Hi chamrong! any chance someone did an outputlookup without append and blew it away?

you could search index=_internal for your lookup name to see if you can identify any searches that may have overwritten it in the search or audit logs...

is it a scheduled search that populates this ?

whats the earliest date you have in the lookup?

As long as you have the data still, you can simply run a search that covers the span of the missing data and rebuild the lookup file....

Be sure to check | inputlookup <yourLookupFile> to verify the users claims!

Re: How to troubleshoot why our lookup that is updated by "outputlookup append=true" missing data added from the last 2 months?

Chamrong — Fri, 09 Jun 2017 21:25:52 GMT

Hey Matt!! Thank for help.
It is a schedule search happened daily.
The earliest is late 2016
The content is coming from the log of certain session

Re: How to troubleshoot why our lookup that is updated by "outputlookup append=true" missing data added from the last 2 months?

mattymo — Fri, 09 Jun 2017 21:51:57 GMT

so if you do an inputlookup do you see the data from late 2016???

http://docs.splunk.com/Documentation/Splunk/6.6.1/SearchReference/Inputlookup

Re: How to troubleshoot why our lookup that is updated by "outputlookup append=true" missing data added from the last 2 months?

maciep — Sat, 10 Jun 2017 13:22:16 GMT

Is it adding new data again since your users brought it to your attention? or is it still broken? If it's still broken, can you run the search manually w/o the outputlookup to see if there are still results?

Also, if it's still broken, is the owner of the saved search still a valid Splunk user? That's a long shot, but I think we've run into it before - user leaves company -> account terminated -> searches stop running (there would be errors in _internal)

Re: How to troubleshoot why our lookup that is updated by "outputlookup append=true" missing data added from the last 2 months?

DalJeanis — Sun, 11 Jun 2017 19:52:40 GMT

Start by checking the output from a single night, see whether it is really appending.

Also, start by checking whether the search that appends the data is running afoul of any subsearch limitations during the intermediate steps.

Then backtrack 1 month at a time, see what the daily jobs were reporting, what errors may have cropped up.

Re: How to troubleshoot why our lookup that is updated by "outputlookup append=true" missing data added from the last 2 months?

Chamrong — Fri, 16 Jun 2017 21:02:37 GMT

We find the root cause. the bulde pushout to the search head by the apps upgreade. To prevent this, we need to add -preserve-lookups true

[5:00]
http://docs.splunk.com/Documentation/Splunk/6.6.0/DistSearch/PropagateSHCconfigurationchanges#Maintain_lookup_files_across_app_upgrades