Splunk_TA_stream and search head clusters (shc)

[sloshburch]

While trying to deploy both Splunk_TA_stream and splunk_app_stream to a SHC, you see the following error and the deploy push fails:

Error while deploying apps to target=https://burch:splunkd-port with members=3: Error while updating app=Splunk_TA_stream on target=https://burch-ip:splunkd-port: Non-200/201 status_code=500; {"messages":[{"type":"ERROR","text":"\n In handler 'localapps': Error installing application: Failed to copy: /opt/splunk/var/run/splunk/bundle_tmp/010fb5c688614565/Splunk_TA_stream to /opt/splunk/etc/apps/Splunk_TA_stream. Error occurred while copying source to destination error=\"Text file busy\" src=\"/opt/splunk/var/run/splunk/bundle_tmp/010fb5c688614565/Splunk_TA_stream/linux_x86_64/bin/streamfwd\" dest=\"/opt/splunk/etc/apps/Splunk_TA_stream/linux_x86_64/bin/streamfwd\""}]}

[sloshburch · Accepted solution]

This is because When splunk_app_stream is pushed by the deployer to the shc members it's process for installing Splunk_TA_stream kicks in. As a result, the deployer fails while trying to push 'Splunk_TA_stream' since it's file handles are open by splunkd on the SHC still deploying Splunk_TA_stream from splunk_app_stream.

I was able to validate and workaround this by disabling all of the inputs listed in splunk_app_stream on the Deployer (which in turn leaves them disabled on the SHC members.

[script://./bin/scripted_inputs/deploy_splunk_ta_stream.py]
    disabled = true

[script://.\\bin\\scripted_inputs\\deploy_splunk_ta_stream.py]
    disabled = true

[script://./bin/scripted_inputs/setup_independent_stream.py]
    disabled = true

[script://.\\bin\\scripted_inputs\\setup_independent_stream.py]
    disabled = true