topic Re: Splunk App for Stream Installation: Missing directories, wire data input and Splunk is running as root in All Apps and Add-ons

Splunk App for Stream Installation: Missing directories, wire data input and Splunk is running as root

rizzo75 — Mon, 28 Sep 2020 17:19:08 GMT

Hi -
I have tried installing the Splunk App for Stream on 2 different Splunk servers(ubuntu 14.04 x86_64) and the experience is the same. I follow install directions: http://docs.splunk.com/Documentation/StreamApp/latest/DeployStreamApp/InstallSplunkAppforStream

Splunk is running as root.
Directories $SPLUNK_HOME/etc/apps/Splunk_TA_stream and $SPLUNK_HOME/etc/deployment-apps/Splunk_TA_stream are not created.
The Wire Data input is not listed.

Any help is appreciated.

Thanks,
Joe

Re: Splunk App for Stream Installation: Missing directories, wire data input and Splunk is running as root

rdeleonsplunk — Mon, 28 Sep 2020 17:19:11 GMT

I've encountered the same issue. It's also interesting to note that Splunk_TA_stream was installed under $SPLUNK_HOME/etc/apps/splunk_app_stream/install/ directory.

Re: Splunk App for Stream Installation: Missing directories, wire data input and Splunk is running as root

mdickey_splunk — Tue, 12 Aug 2014 22:41:36 GMT

Hi Joe,

Did you install using Splunk's web interface or by just uncompressing the file into $SPLUNK_HOME/etc/apps? You do need to restart your splunk server for it to create the Splunk_TA_stream directory, an add-on that provides the Wire data input. The web UI prompts to do this for you, but installing via command line requires a manual restart using "$SPLUNK_HOME/bin/splunk restart". The script that does this should also create a log file $SPLUNK_HOME/var/log/splunk/stream_installer.log; if there is a problem it may indicate what is wrong.

If all else fails, you can also just manually copy the contents from $SPLUNK_HOME/etc/apps/splunk_app_stream/install/Splunk_TA_stream into $SPLUNK_HOME/etc/apps/Splunk_TA_stream and restart splunk. It should definitely pick things up after that.

Take care,

-Mike

Re: Splunk App for Stream Installation: Missing directories, wire data input and Splunk is running as root

rdeleonsplunk — Mon, 28 Sep 2020 17:19:14 GMT

Mike,

I manually copied Splunk_TA_stream directory into $SPLUNK_HOME/etc/apps and i'm now able to see the "Wire data" option in the Data Inputs. Nice!

However, when I click "Wire Data" I don't see "streamfwd" in the list. In fact, there are no items displayed on the Data Inputs > Wire Data page.

I also checked $SPLUNK_HOME/var/log/splunk/stream_installer.log but this file does not exist in the log directory.

Any ideas what I'm missing? Thanks!

Re: Splunk App for Stream Installation: Missing directories, wire data input and Splunk is running as root

rizzo75 — Mon, 28 Sep 2020 17:19:17 GMT

Thanks for the response.

I installed the app via the web interface.

$SPLUNK_HOME/var/log/splunk/stream_installer.log does not exist.

I just tried installing from the command line with the same results.

I do notice this in the splunkd.log: http://pastebin.com/MDuHXWqK

Thanks,
Joe

Re: Splunk App for Stream Installation: Missing directories, wire data input and Splunk is running as root

rdeleonsplunk — Tue, 12 Aug 2014 23:14:15 GMT

P.S. I still don't see "streamfwd" in the Wire Data settings page but I just ran the query -> sourcetype="stream:http"

and i'm now seeing HTTP data stream! awesome! 🙂

Re: Splunk App for Stream Installation: Missing directories, wire data input and Splunk is running as root

rizzo75 — Mon, 28 Sep 2020 17:19:20 GMT

I also manually copied the Splunk_TA_stream directory to the apps directory and did not see a streamfwd input.

Joe

Re: Splunk App for Stream Installation: Missing directories, wire data input and Splunk is running as root

mdickey_splunk — Wed, 13 Aug 2014 00:02:39 GMT

@rizzo75, thanks for that post. I just reproduced and can confirm we seem to have a bug in 6.0.0 where the depedency_manager.py script (which deploys Splunk_TA_stream) conflicts with a similarly-named script provided by the App for Unix.

You can work-around this by manually copying Splunk_TA_stream from $SPLUNK_HOME/etc/apps/splunk_app_stream/install/Splunk_TA_stream into $SPLUNK_HOME/etc/apps/.

Please note that this script also creates the default "streamfwd" Wire data input, so when you manually copy the TA directory, you will also need to create a new Wire data input using the Splunk UI. Normally, you should be able to just cut and paste the example URL into the single-field input box.

Re: Splunk App for Stream Installation: Missing directories, wire data input and Splunk is running as root

dstaulcu — Wed, 13 Aug 2014 00:54:16 GMT

Had the same problem too. Referred to Before You Deploy > Deployment Requirements section of DeployStreamApp documentation and found that Splunk instances on Windows are not supported. Also noted that non-enterprise versions of Splunk are also not supported.

Re: Splunk App for Stream Installation: Missing directories, wire data input and Splunk is running as root

rdeleonsplunk — Wed, 13 Aug 2014 15:36:38 GMT

@mdickey_splunk

thanks for the help. i finally (almost) got it to work. i'm now seeing the streamfwd logs. and i saw this message: "No capture devices found (must be root/Administrator)"
better contact our sysadmins to give streamfwd root access 🙂

Re: Splunk App for Stream Installation: Missing directories, wire data input and Splunk is running as root

shandman — Wed, 13 Aug 2014 21:06:58 GMT

Having the same issue. Have spent hours on this. Would sure be nice to get a simple documented process to fix this.

Re: Splunk App for Stream Installation: Missing directories, wire data input and Splunk is running as root

mdickey_splunk — Mon, 28 Sep 2020 17:19:44 GMT

@shandman I'm sorry you are experiencing problems with this. Do you have App for Unix installed and see similar errors in your splunkd.log file about dependency_manager.py? If so then the step-by-step instructions documented in the troubleshooting guide (http://docs.splunk.com/Documentation/StreamApp/6.0/DeployStreamApp/Troubleshooting#Splunk_TA_stream_and_Wire_Data_mod_input_not_appearing_after_install) should fix the problem. We've also updated the installation documentation to refer to this troubleshooting article, and plan to have a new release including the fix soon.

Re: Splunk App for Stream Installation: Missing directories, wire data input and Splunk is running as root

shandman — Thu, 14 Aug 2014 15:26:53 GMT

ty@mdickey_splunk

Yes, I had the App for Unix installed. I had tried going through the documentation in the link you refer to "before" finding this thread. I ended up wrestling it with a few more hours.

Copied files per instructions didn't work.
Removing / deleting Unix app didn't work.
Reinstalled stream app. Copied directory manually. / worked

Thanks again.

Re: Splunk App for Stream Installation: Missing directories, wire data input and Splunk is running as root

mdickey_splunk — Sat, 16 Aug 2014 00:42:35 GMT

Splunk App for Stream 6.0.1 has been released! This build fixes several problems regarding the initial configuration of the wire data input. You can download it here:

http://apps.splunk.com/app/1809/

Release notes here:

http://docs.splunk.com/Documentation/StreamApp/6.0.1/ReleaseNotes/FixedProblems

Please let me know if you experience any problems with the wire data input using version 6.0.1.

Thanks,

-Mike

Re: Splunk App for Stream Installation: Missing directories, wire data input and Splunk is running as root

Akili — Thu, 19 Feb 2015 14:31:33 GMT

stream installer log

[root@splunk splunk]# cat stream_installer.log
2015-02-12 16:20:53,667 [INFO] Splunk App for Stream Dependency Manager: Starting...
2015-02-12 16:24:11,975 [INFO] Splunk App for Stream Dependency Manager: Starting...
2015-02-16 10:31:14,928 [INFO] Splunk App for Stream Dependency Manager: Starting...
2015-02-16 14:46:10,484 [INFO] Splunk App for Stream Dependency Manager: Starting...
2015-02-17 10:25:51,415 [INFO] Splunk App for Stream Dependency Manager: Starting...
2015-02-17 14:50:41,790 [INFO] Splunk App for Stream Dependency Manager: Starting...
2015-02-17 15:02:53,880 [INFO] Splunk App for Stream Dependency Manager: Starting...
2015-02-17 15:05:35,691 [INFO] Splunk App for Stream Dependency Manager: Starting...
2015-02-17 15:25:11,921 [INFO] Splunk App for Stream Dependency Manager: Starting...
2015-02-18 15:41:06,928 [INFO] Splunk App for Stream Dependency Manager: Starting...
2015-02-19 09:23:44,605 [INFO] Splunk App for Stream Dependency Manager: Starting...
2015-02-19 15:19:53,318 [INFO] Splunk App for Stream Dependency Manager: Starting...
2015-02-19 15:23:15,402 [INFO] Splunk App for Stream Dependency Manager: Starting...
[root@splunk splunk]#

Re: Splunk App for Stream Installation: Missing directories, wire data input and Splunk is running as root

Akili — Thu, 19 Feb 2015 14:36:19 GMT

i didn't have the app for unix installed and still got the same problems

Re: Splunk App for Stream Installation: Missing directories, wire data input and Splunk is running as root

Akili — Thu, 19 Feb 2015 14:55:40 GMT

me too. do not see a streamfwd input

Re: Splunk App for Stream Installation: Missing directories, wire data input and Splunk is running as root

Akili — Mon, 28 Sep 2020 18:58:58 GMT

[root@splunk Splunk_TA_stream]# more streamfwd.log
2015-02-19 15:37:48 INFO 140253906425664 stream.CaptureServer - Found DataDirecto
ry: /opt/splunk/etc/apps/Splunk_TA_stream/data
2015-02-19 15:37:48 INFO 140253906425664 stream.CaptureServer - Found UIDirectory
: /opt/splunk/etc/apps/Splunk_TA_stream/ui
2015-02-19 15:37:48 INFO 140608789518144 stream.CaptureServer - Found DataDirecto
ry: /opt/splunk/etc/apps/Splunk_TA_stream/data
2015-02-19 15:37:48 INFO 140608789518144 stream.CaptureServer - Found UIDirectory
: /opt/splunk/etc/apps/Splunk_TA_stream/ui
2015-02-19 15:37:48 INFO 140608789518144 stream.CaptureServer - Loaded configurat
ion file: /opt/splunk/etc/apps/Splunk_TA_stream/local/streamfwd.xml
2015-02-19 15:37:48 ERROR 140608789518144 stream.CaptureServer - Unable to ping s
erver (b37e1dc2-6377-4308-8556-02d2da6543ca): Unable to establish connection to localhost: Connection refuse
d

Re: Splunk App for Stream Installation: Missing directories, wire data input and Splunk is running as root

Akili — Thu, 19 Feb 2015 15:08:36 GMT

while adding streamfwd from data inputs wire data

Encountered the following error while trying to save: In handler 'streamfwd': The script returned with exit status 2.

Re: Splunk App for Stream Installation: Missing directories, wire data input and Splunk is running as root

rphillips_splk — Tue, 29 Sep 2020 07:01:20 GMT

I encountered the same error ( Encountered the following error while trying to save: In handler 'streamfwd': The script returned with exit status 2.) when trying to edit the modular input streamfwd (more settings) to set it to another index.
splunkd showed this:
08-18-2015 23:57:46.463 -0700 ERROR ModularInputs - Argument validation for scheme=streamfwd: killing process, because executing it took too long (over 30000 msecs).
08-18-2015 23:57:46.465 -0700 INFO ModularInputs - Argument validation for scheme=streamfwd: script running failed (killed by signal 9: Killed: 9).

I initially untarred the splunk_app_stream.tar file and copied it into etc/apps/ and restarted splunk for the first install which led me to the error.

To fix the issue I removed the Splunk_TA_stream and the splunk_app_stream , restarted splunk then installed from the web UI under Apps>Find More Apps . I then enabled the modular input through the web UI (Settings>Data Inputs> Wire Data > streamfwd - enable.

To see http data I went to the Splunk App for Stream from the app menu and enabled the http protocol. Then did a search for index=* source=stream* and see data now.

My system was a standalone server so fwdr/SH/IDX all in one.