Deployment Architecture

Deployment apps not reloading

mxanareckless
Path Finder

Here are the contents of the /opt/splunk/etc/deployment-apps:

[splunk@splunk deployment-apps]$ ls
README  Splunk_TA_nix  Splunk_TA_windows
[splunk@splunk deployment-apps]$ pwd
/opt/splunk/etc/deployment-apps

However upon running either splunk reload deploy-server or splunk restart, while no errors are being generated, the deployment apps displayed on the web UI are not updating:

mxanareckless_0-1617309967364.png

The entire path /opt/splunk/etc/deployment-apps and its subfolders are owned by the splunk user.

Here are the results when running splunk reload deploy-server -debug:

[splunk@splunk deployment-apps]$ splunk reload deploy-server -debug > reload
Will setenv SPLUNK_CLI_DEBUG to "v".
In check_and_set_splunk_os_user(): In env found SPLUNK_OS_USER var: "splunk"
In cmd_rc
Will exec (detach=no): LDAPCONF=/disk2/splunk/etc/openldap/ldap.conf HOSTNAME=splunk.cdc.health.local SPLUNK_SERVER_NAME=Splunkd SPLUNK_HOME=/disk2/splunk USER=splunk LD_LIBRARY_PATH=/disk2/splunk/lib:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64/jli PATH=/disk2/splunk/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/splunk/.local/bin:/home/splunk/bin:/disk2/splunk/bin PWD=/opt/splunk/etc/deployment-apps SPLUNK_DB=/export/intindx NODE_PATH=/disk2/splunk/lib/node_modules SPLUNK_OS_USER=splunk /disk2/splunk/bin/splunkd btool web list settings --no-log
Will exec (detach=no): LDAPCONF=/disk2/splunk/etc/openldap/ldap.conf HOSTNAME=splunk.cdc.health.local SPLUNK_SERVER_NAME=Splunkd SPLUNK_HOME=/disk2/splunk USER=splunk LD_LIBRARY_PATH=/disk2/splunk/lib:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64/jli PATH=/disk2/splunk/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/splunk/.local/bin:/home/splunk/bin:/disk2/splunk/bin PWD=/opt/splunk/etc/deployment-apps SPLUNK_DB=/export/intindx NODE_PATH=/disk2/splunk/lib/node_modules SPLUNK_OS_USER=splunk /disk2/splunk/bin/splunkd btool server list general --no-log
Will exec (detach=no): LDAPCONF=/disk2/splunk/etc/openldap/ldap.conf HOSTNAME=splunk.cdc.health.local SPLUNK_SERVER_NAME=Splunkd SPLUNK_HOME=/disk2/splunk USER=splunk LD_LIBRARY_PATH=/disk2/splunk/lib:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64/jli PATH=/disk2/splunk/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/splunk/.local/bin:/home/splunk/bin:/disk2/splunk/bin PWD=/opt/splunk/etc/deployment-apps SPLUNK_DB=/export/intindx NODE_PATH=/disk2/splunk/lib/node_modules SPLUNK_OS_USER=splunk /disk2/splunk/bin/splunkd btool server list kvstore --no-log
Will exec (detach=no): LDAPCONF=/disk2/splunk/etc/openldap/ldap.conf HOSTNAME=splunk.cdc.health.local SPLUNK_SERVER_NAME=Splunkd SPLUNK_HOME=/disk2/splunk USER=splunk LD_LIBRARY_PATH=/disk2/splunk/lib:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64/jli PATH=/disk2/splunk/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/splunk/.local/bin:/home/splunk/bin:/disk2/splunk/bin PWD=/opt/splunk/etc/deployment-apps SPLUNK_DB=/export/intindx NODE_PATH=/disk2/splunk/lib/node_modules SPLUNK_OS_USER=splunk /disk2/splunk/bin/splunkd btool server list dfs --no-log
dynamic conf init: httpport=8000 mgmtHostPort=8089 bind_ip= rest_hostname=127.0.0.1 kvStorePort=8191
base splunkdURL from splunkd: https://127.0.0.1:8089
Will setenv SPLUNK_CLI_DEBUG to "v".
cmd: reload
obj: deploy-server
cli args:
In build_full_rest_url(): Composing URL from base=https://127.0.0.1:8089 + relative=/static/splunkrc_cmds.xml
In build_full_rest_url(): Composed URL=https://127.0.0.1:8089/static/splunkrc_cmds.xml
In make_simple_rest_call_online(): using_basic_auth=0
In make_simple_rest_call_online(): [Re-]Initialized HTTP request headers:
        <none>
In make_simple_rest_call_online(): HTTP request response_code=200
In make_simple_rest_call_online(): HTTP response headers:
        ETag: "s-t2y1XwAAAAD4ZQsAAAAAAJ8CCgAAAAAANsQDAAAAAAA"
        Date: Thu, 01 Apr 2021 20:44:37 GMT
        Expires: Thu, 01 Apr 2021 21:44:37 GMT
        Cache-Control: max-age=3600
        Content-Type: text/xml
        X-Content-Type-Options: nosniff
        Last-Modified: Wed, 18 Nov 2020 18:49:27 GMT
        Content-Length: 246838
        Connection: Keep-Alive
        X-Frame-Options: SAMEORIGIN
        Server: Splunkd
Online REST call to /static/splunkrc_cmds.xml returned 0
Visited node: deploy-server
Entered node:deploy-server/common
Entered node:deploy-server/cmd

Metadata:
cmd: reload
obj: deploy-server
rel_url: (no value)
eai_id: (no value)
implied_arg_name: (no value)
action: 0
args map count: 0
args map:
default args count: 0
initial args count: 0
hooks count: 0
object help
command array count: 0
In call_pcl, will exec_python
Will exec (detach=no): HOSTNAME=splunk.cdc.health.local USER=splunk PATH=/disk2/splunk/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/splunk/.local/bin:/home/splunk/bin:/disk2/splunk/bin PWD=/opt/splunk/etc/deployment-apps SPLUNK_HOME=/disk2/splunk SPLUNK_DB=/export/intindx SPLUNK_SERVER_NAME=Splunkd SPLUNK_OS_USER=splunk NODE_PATH=/disk2/splunk/lib/node_modules PYTHONPATH= LD_LIBRARY_PATH=/disk2/splunk/lib:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64/jli LDAPCONF=/disk2/splunk/etc/openldap/ldap.conf /disk2/splunk/bin/python3.7 -u /disk2/splunk/lib/python3.7/site-packages/splunk/clilib/cli.py reload deploy-server
DEBUG (cli) Running in debug mode.

DEBUG (cli) Command: reload
DEBUG (cli) Subcmd:  deploy-server
DEBUG (cli) Begin parsed arguments:
DEBUG (cli)
DEBUG (cli) End parsed arguments.


DEBUG (cli) noAuthReq: False
DEBUG (cli) username: None
DEBUG (cli) password: None
DEBUG (cli_common) Running btool for 'web.conf'.
WARNING (cli_common) btool returned something in stderr: 'Will exec (detach=no): HOSTNAME=splunk.cdc.health.local USER=splunk PATH=/disk2/splunk/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/splunk/.local/bin:/home/splunk/bin:/disk2/splunk/bin PWD=/opt/splunk/etc/deployment-apps SPLUNK_HOME=/disk2/splunk SPLUNK_DB=/export/intindx SPLUNK_SERVER_NAME=Splunkd SPLUNK_OS_USER=splunk NODE_PATH=/disk2/splunk/lib/node_modules LD_LIBRARY_PATH=/disk2/splunk/lib:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64/jli LDAPCONF=/disk2/splunk/etc/openldap/ldap.conf /disk2/splunk/bin/splunkd btool web list
'
DEBUG (cli) Contents of ./splunk/authToken file: <auth><username>splunkadmin</username><sessionkey></sessionkey></auth>
DEBUG (cli) authInfo: 
DEBUG (rcUtils) cmd: reload, obj: deploy-server, restArgList: {'authstr':', 'owner': 'splunkadmin'}
DEBUG (rcUtils) endpoint: deployments
DEBUG (rcUtils) layeredFind:default_eai_parms:
DEBUG (rcUtils) authstr poped from argList
DEBUG (rcUtils) layeredFind:app_context:
DEBUG (rcUtils) namespace:
DEBUG (rcUtils) layeredFind:uri: /deployment/server/config/_reload
DEBUG (rcUtils) layeredFind:eai_id:
DEBUG (rcUtils) layeredFind:required:
DEBUG (rcUtils) eai_key_list: []
DEBUG (rcUtils) eai_key_list: []
DEBUG (rcUtils) layeredFind:args:
DEBUG (rcUtils) layeredFind:prehooks:
DEBUG (rcUtils) after prehooks, eaiArgsList: {}
DEBUG (rcUtils) Before buildEndpoint uri: /deployment/server/config/_reload
DEBUG (rcUtils) Before buildEndpoint entityName:
DEBUG (rcUtils) uri: /services/deployment/server/config/_reload
DEBUG (rcUtils) layeredFind:type: edit
DEBUG (rcUtils) eaiArgsList: {}
DEBUG (rcUtils) In sanitizeArgs: target: , argsMap: {}, argsDict: {}
DEBUG (rcUtils) In sanitizeArgs: target: , argsMap: {}, argsDict: {}
DEBUG (rcUtils) In sanitizeArgs: target: /deployment/server/config/_reload, argsMap: {}, argsDict: {}
DEBUG (rcUtils) In sanitizeArgs: target: /deployment/server/config/_reload, argsMap: {}, argsDict: {}
DEBUG (rcUtils) postargs: {}
DEBUG (rcUtils) getargs: {}
DEBUG (__init__) simpleRequest > POST https://127.0.0.1:8089/services/deployment/server/config/_reload [] sessionSource=direct timeout=30
DEBUG (cli_common) Running btool for 'server.conf'.
WARNING (cli_common) btool returned something in stderr: 'Will exec (detach=no): HOSTNAME=splunk.cdc.health.local USER=splunk PATH=/disk2/splunk/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/splunk/.local/bin:/home/splunk/bin:/disk2/splunk/bin PWD=/opt/splunk/etc/deployment-apps SPLUNK_HOME=/disk2/splunk SPLUNK_DB=/export/intindx SPLUNK_SERVER_NAME=Splunkd SPLUNK_OS_USER=splunk NODE_PATH=/disk2/splunk/lib/node_modules LD_LIBRARY_PATH=/disk2/splunk/lib:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64/jli LDAPCONF=/disk2/splunk/etc/openldap/ldap.conf /disk2/splunk/bin/splunkd btool server list
'
DEBUG (__init__) simpleRequest < server responded status=200 responseTime=0.1081s
DEBUG (rcUtils) In checkStatus: type: edit, server_response: {'date': 'Thu, 01 Apr 2021 20:44:37 GMT', 'expires': 'Thu, 26 Oct 1978 00:00:00 GMT', 'cache-control': 'no-store, no-cache, must-revalidate, max-age=0', 'content-type': 'text/xml; charset=UTF-8', 'x-content-type-options': 'nosniff', 'content-length': '3828', 'vary': 'Authorization', 'connection': 'Close', 'x-frame-options': 'SAMEORIGIN', 'server': 'Splunkd', 'status': '200'}
DEBUG (rcDisplay) In displayDeployment: kwargs: {'cmd': 'reload', 'obj': 'deploy-server', 'type': 'edit', 'serverResponse': {'date': 'Thu, 01 Apr 2021 20:44:37 GMT', 'expires': 'Thu, 26 Oct 1978 00:00:00 GMT', 'cache-control': 'no-store, no-cache, must-revalidate, max-age=0', 'content-type': 'text/xml; charset=UTF-8', 'x-content-type-options': 'nosniff', 'content-length': '3828', 'vary': 'Authorization', 'connection': 'Close', 'x-frame-options': 'SAMEORIGIN', 'server': 'Splunkd', 'status': '200'}, 'serverContent': b'<?xml version="1.0" encoding="UTF-8"?>\n<!--This is to override browser formatting; see server.conf[httpServer] to disable. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .-->\n<?xml-stylesheet type="text/xml" href="/static/atom.xsl"?>\n<feed xmlns="http://www.w3.org/2005/Atom" xmlns:s="http://dev.splunk.com/ns/rest" xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">\n  <title>deploymentserver</title>\n  <id>https://127.0.0.1:8089/services/deployment/server/config</id>\n  <updated>2021-04-01T20:44:37+00:00</updated>\n  <generator build="152fb4b2bb96" version="8.0.6"/>\n  <author>\n    <name>Splunk</name>\n  </author>\n  <link href="/services/deployment/server/config/_reload" rel="_reload"/>\n  <link href="/services/deployment/server/config/attributesUnsupportedInUI" rel="attributesUnsupportedInUI"/>\n  <link href="/services/deployment/server/config/authentication" rel="authentication"/>\n  <link href="/services/deployment/server/config/listIsDisabled" rel="listIsDisabled"/>\n  <opensearch:totalResults>1</opensearch:totalResults>\n  <opensearch:itemsPerPage>30</opensearch:itemsPerPage>\n  <opensearch:startIndex>0</opensearch:startIndex>\n  <s:messages/>\n  <entry>\n    <title>config</title>\n    <id>https://127.0.0.1:8089/services/deployment/server/config/config</id>\n    <updated>1970-01-01T00:00:00+00:00</updated>\n    <link href="/services/deployment/server/config/config" rel="alternate"/>\n    <author>\n      <name>system</name>\n    </author>\n    <link href="/services/deployment/server/config/config" rel="list"/>\n    <link href="/services/deployment/server/config/config/_reload" rel="_reload"/>\n    <link href="/services/deployment/server/config/config" rel="edit"/>\n    <content type="text/xml">\n      <s:dict>\n        <s:key name="currentDownloads">0</s:key>\n        <s:key name="disabled">0</s:key>\n        <s:key name="eai:acl">\n          <s:dict>\n            <s:key name="app"></s:key>\n            <s:key name="can_list">1</s:key>\n            <s:key name="can_write">1</s:key>\n            <s:key name="modifiable">0</s:key>\n            <s:key name="owner">system</s:key>\n            <s:key name="perms">\n              <s:dict>\n                <s:key name="read">\n                  <s:list>\n                    <s:item>admin</s:item>\n                    <s:item>splunk-system-role</s:item>\n                  </s:list>\n                </s:key>\n                <s:key name="write">\n                  <s:list>\n                    <s:item>admin</s:item>\n                    <s:item>splunk-system-role</s:item>\n                  </s:list>\n                </s:key>\n              </s:dict>\n            </s:key>\n            <s:key name="removable">0</s:key>\n            <s:key name="sharing">system</s:key>\n          </s:dict>\n        </s:key>\n        <s:key name="loadTime">1617309877</s:key>\n        <s:key name="repositoryLocation">$SPLUNK_HOME/etc/deployment-apps</s:key>\n      </s:dict>\n    </content>\n  </entry>\n</feed>\n', 'sessionKey': '', 'eaiArgsList': {}}

 

0 Karma
1 Solution

scelikok
Champion

Hi @mxanareckless,

In your debug log SPLUNK_HOME is shown as /disk2/splunk.

You need to manage the deployment apps under /disk2/splunk/etc/deployment-apps.

 

If this reply helps you an upvote is appreciated.

View solution in original post

scelikok
Champion

Hi @mxanareckless,

In your debug log SPLUNK_HOME is shown as /disk2/splunk.

You need to manage the deployment apps under /disk2/splunk/etc/deployment-apps.

 

If this reply helps you an upvote is appreciated.

View solution in original post

Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!