Splunk 6 with Splunk for Unix Add-on: Setup Dashboard goes 404 not found

[pthompson]

New to splunk... getting to know my way around.

Installed the Splunk for Unix Add-on App, but the Dashboard doesnt work. Navigating to the Dashboard Setup gets you a 404 error.

The app is collecting stats, just the Dashboard is broken. Need some pointers where to start looking.

On Linux Centos 2.6 64 bit.

[yoho · Accepted solution]

Followup on this: In the meantime, we've recovered our enterprise entitlement. I've found the problem for this issue: I've been playing too much with local.meta to limit permissions and even admin was not able to "write" (the config). It's now fixed.

[yoho]

Actually got again the problem with another platform. My problem was not playing with permissions, it was more subtle than that. I had a custom app with a "bin/unix" directory (copy from a previous unix app), it was sufficient for python to look for several libraries under this path which it did not find. Error message in the web_service.log led me to the way to solve it (the message was saying it was unable to load a library)

[araitz]

Can you look in $SPLUNK_HOME/var/log/splunk/web_service.log? There should be an error there describing what happened to the unixsetup controller. You can also open a support ticket so we can get a diag and see what is going on.

[yoho]

I also run the same OS but I don't think it's tied to the OS.

I've read somewhere else that it was because splunk was running as root and indeed, by accident, I've been running splunk as root. I stopped splunk, change ownership of all files back to "splunk:splunk" and start it again. Unfortunately, it was still not running so I'm not sure this is the problem (I mention it here because it may be the problem of other people reading this answer).

So I still don't have an answer to this problem but found out that the actual missing page (404) is http://splunkinstance:8000/splunkroot/en-GB/custom/splunk_app_for_nix/unixsetup/SA-nix/show. I've found the server side included HTML responsible for this link but it looks like the "custom" branch is on purpose, although there is no such "custom" directory on disk. So I'm stuck 😞

Note: I run splunk 5.0.5, and not 6 like the title says...

[araitz]

I understand. It is difficult to help with the issue here because I can't reproduce and can't investigate your logs. Please keep me updated.

[yoho]

Araitz, for some reason my entitlement (enterprise support) has been "lost" and I am now only allowed to open community cases. I'm trying to solve this issue first because it's critical to my business. Once I get enterprise support, the first case I'll open is about this problem.

[araitz]

Any results from the support case? Can you let me know the case number?

[yoho]

Ok. I'll do. Thanks.

[araitz]

Yes, we have many customers running on 5.x and 6.x. Those errors indicate that the controllers unixsetup, unixheadlines, unixalertsconfig, and unixalertevents are unable to initialize because they cannot import python files from the app's /bin directory. It would be great if you could open a support case so we could help you figure out what is going on in your environment.

[yoho]

Didn't open a support case but I found the following errors when starting splunk in web_service.log. Are you sure the app should work with Splunk 5 ? :
cannot load specified module unixalertevents in app splunk_app_for_nix: No module named util.timesince
cannot load specified module unixalertsconfig in app splunk_app_for_nix: No module named saved_search
cannot load specified module unixheadlines in app splunk_app_for_nix: No module named util.timesince
cannot load specified module unixsetup in app splunk_app_for_nix: No module named macro

[araitz]

Any update? Were you able to open a support case?

[yoho]

Ok, then I can continue troubleshooting my problem, thanks for the info !

[araitz]

The /custom path is served by our app server. Any custom controller within an app will be mounted there if there is a [custom:] entry in web.conf.

The actual file lives in the unix app under appserver/controllers/unixsetup.py.

See: http://dev.splunk.com/view/SP-CAAADS2