I had the same issue which started after a recent upgrade when I mistakenly restored a backup from a search head to the deployer and then deployed that bundle using the full push mode. This created a default.meta file in each search head app which - despite not having a removable property in each lookup's stanza - changed the endpoint "removable" property to 0. curl -k -u admin:password https[:]//localhost:8089/servicesNS/<lookup owner>/<app name>/data/lookup-table-files/<lookup file.csv> <?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xml" href="/static/atom.xsl"?>
<feed xmlns="http[:]//www.w3.org/2005/Atom" xmlns:s="http[:]//dev.splunk.com/ns/rest" xmlns:opensearch="http[:]//a9.com/-/spec/opensearch/1.1/">
<title>lookup-table-files</title>
<id>https[:]//localhost:8089/servicesNS/<lookup owner>/<app name>/data/lookup-table-files</id>
<updated>2021-08-18T15:05:43-06:00</updated>
<generator build="*****" version="8.2.1"/>
<author>
<name>Splunk</name>
</author>
<link href="/servicesNS/<lookup owner>/<app name>/data/lookup-table-files/_new" rel="create"/>
<link href="/servicesNS/<lookup owner>/<app name>/data/lookup-table-files/_reload" rel="_reload"/>
<link href="/servicesNS/<lookup owner>/<app name>/data/lookup-table-files/_acl" rel="_acl"/>
<opensearch:totalResults>1</opensearch:totalResults>
<opensearch:itemsPerPage>30</opensearch:itemsPerPage>
<opensearch:startIndex>0</opensearch:startIndex>
<s:messages/>
<entry>
<title><lookup file.csv></title>
<id>https[:]//localhost:8089/servicesNS/<lookup owner>/<app name>/data/lookup-table-files/<lookup file.csv></id>
<updated>2021-08-18T12:50:10-06:00</updated>
<link href="/servicesNS/<lookup owner>/<app name>/data/lookup-table-files/<lookup file.csv>" rel="alternate"/>
<author>
<name><lookup owner></name>
</author>
<link href="/servicesNS/<lookup owner>/<app name>/data/lookup-table-files/<lookup file.csv>" rel="list"/>
<link href="/servicesNS/<lookup owner>/<app name>/data/lookup-table-files/<lookup file.csv>/_reload" rel="_reload"/>
<link href="/servicesNS/<lookup owner>/<app name>/data/lookup-table-files/<lookup file.csv>" rel="edit"/>
<!-- The following two links were missing from the response; pulled from a working lookup
<link href="/servicesNS/<lookup owner>/<app name>/data/lookup-table-files/<lookup file.csv>" rel="remove"/>
<link href="/servicesNS/<lookup owner>/<app name>/data/lookup-table-files/<lookup file.csv>/move" rel="move"/> -->
<content type="text/xml">
<s:dict>
<s:key name="disabled">0</s:key>
<s:key name="eai:acl">
<s:dict>
<s:key name="app"><app name></s:key>
<s:key name="can_change_perms">1</s:key>
<s:key name="can_list">1</s:key>
<s:key name="can_share_app">1</s:key>
<s:key name="can_share_global">1</s:key>
<s:key name="can_share_user">0</s:key>
<s:key name="can_write">1</s:key>
<s:key name="modifiable">1</s:key>
<s:key name="owner"><lookup owner></s:key>
<s:key name="perms">
<s:dict>
<s:key name="read">
<s:list>
<s:item>*</s:item>
</s:list>
</s:key>
<s:key name="write">
<s:list>
<s:item>power</s:item>
</s:list>
</s:key>
</s:dict>
</s:key>
<!-- Here is the problem property: -->
<s:key name="removable">0</s:key>
<s:key name="sharing">app</s:key>
</s:dict>
</s:key>
<s:key name="eai:appName"><app name></s:key>
<s:key name="eai:attributes">
<s:dict>
<s:key name="optionalFields">
<s:list/>
</s:key>
<s:key name="requiredFields">
<s:list>
<s:item>eai:data</s:item>
</s:list>
</s:key>
<s:key name="wildcardFields">
<s:list/>
</s:key>
</s:dict>
</s:key>
<s:key name="eai:data"><![CDATA[/opt/splunk/etc/apps/<app name>/lookups/<lookup file.csv>]]></s:key>
<s:key name="eai:userName"><lookup owner></s:key>
</s:dict>
</content>
</entry>
</feed> After reading mbadhusha_splun's answer, I realized SOME config was overriding the desired config, but there is no lookups.conf. So, I tried renaming /opt/splunk/etc/apps/<app name>/metadata/default.meta on the impacted apps to default.meta_bak on each search head and then restarting them. This resolved our problem! I have some cleanup to do, but the lessons for me were A) change push deploy mode with CAUTION and B) default.meta somehow overrides local.meta.
... View more
