Best practice to ship a file-based lookup in an app, so users can edit it without subsequent app upgrades clobbering those edits.

[sideview]

The Problem:
Unlike with conf, I don't think lookups have any default/local layering. So if your app has one or more lookups that are designed to be customized by users, you have a paradox. On the one hand you need to ship the lookup file with a placeholder row in it, or else the Splunk UI will display a lot of errors when the lookup(s) fail. On the other hand you can't actually ship the lookup file in the app because then when the customer upgrades to a maintenance release of the app, the shipped lookup file will unexpectedly overwrite all of their local edits.

I'd love to hear what ideas people have had in the field to work around this problem. Plus maybe all this time there is some secret way to ship a default lookup or an ftr-generated lookup or something.

What I have done in this situation, is clunky and subject to some problems. The app ships with the lookup missing, and then secretly whenever the homepage is hit the app runs a search like this.

| inputlookup groups | append [|stats count | eval number="aaaaa" | eval group="PLACEHOLDER_GROUP" | eval name="PLACEHOLDER_NAME"] | dedup number group name | sort 0 number | streamstats count | search (name="PLACEHOLDER_NAME" AND count="1") OR NOT name="PLACEHOLDER_NAME" | fields number group name | outputlookup groups

Which, if there is no lookup there, it will create one with a single row "aaaaa,PLACEHOLDER_NAME,PLACEHOLDER_GROUP". And if there is a lookup there, it will read the whole thing in, sort it by number and then write it out again, So, it does nothing, except for use a bunch of system resources for no reason.

Bizarre, clunky, subject to problems, but it generally solves the problem.

problems with that solution
- if users use the 6.X navigation to skip over the homepage pageview entirely but it generally solves the problem.
- doesn't help you for any apps/TA's that are designed to be pushed out to indexers.
- when the lookup is very large, the resource hit can get pretty intense from hauling the whole lookup off disk and then back again.
- clearly evil.

And if you have a use case where you want to have some number of "default" rows, but not clobber on subsequent upgrades, you could ship a second lookup suffixed _default, and then in your semi-evil ftr inputlookup+outputlookup search, manually merge the _default lookup into the main one.

Again - curious if anyone has found a better way to deal with the situation of shipping customer-editable lookups.

[woodcock]

Generally the solution is to deploy all of the AUTHOR's Knowledge Objects (including lookups) in the default directory which the users cannot modify. When a user modifies this, it is written to the local directory which has precedence over default. When an app update is deployed, the default is overwritten but the local copy is not. What is wrong with this?

Another option is to store your lookups as a wiki that is community-edited so everyone is always up to date with the same/latest information:
https://splunkbase.splunk.com/app/635/

[sideview]

the csv files for file-based lookups go in the "lookups" directory, thus are neither in default nor local. The transforms.conf and props.conf keys that configure the use of that lookup might be in default or local and indeed the layering mechanism works for those. For the lookup file itself however, there is no analog of default vs local. Thus if an app ships the actual file in the lookup directory, then on installing any maintenance release of that app, the customer will clobber all of their local updates or additions to the file.

[woodcock]

Good point; go with the web-based wiki lookup.