Yo Splunkers, I am a Splunk 6.3 user, supporting users running on Mac OS X 10.11.1. Yesterday I spent about 1.5 hours investigating and determining a work-around to "Splunk's Little Helper" failing to help on startup and shutdown. Hopefully this helps others who may be experiencing the same issue. The problem and work-around has been repeatedly tested and has shown itself to be effective 100% of the times I tested it. Here are the details that I provided to my fellow users: Team, Hamid contacted me last night because Parin was having an issue with Splunk. We worked for over an hour and debugged what is going on. NOTE: This may be a problem specific to Splunk 6.3 on Mac OSX 10.11.1 (El Capitan with latest patches applied) so if your splunk installation is working OK you can ignore this. It may also be an issue with the specific hardware Parin and I are running: Macbook Pro / i7 / Quad core / late 2011 (model 8.2) / 16 or 8GB of ram. "Splunk's Little Helper" has an issue on the above referenced version of the MAC OS / hrdwr config: it ain't helpin! It does not start or stop Splunk correctly and Splunk hangs. There seems to be some kind of race condition occurring that results in "SLH" hanging and not properly starting the Splunk daemon (the MAC OS/LUNIX equivalent of the WIN service) and / or the Web server component that is used for the GUI. Symptoms: 1) You are starting or stopping splunk. 2) You double click the "Splunk.app" on your desktop or application folder. 3) SLH shows it's GUI and you select either "Start and Show" or "Start only". 4) Depending on which option you selected you see either: "Start and Show Splunk" - A blank browser window and no splunk GUI displayed (the start and show option) and a terminal window with just the BASH prompt and no screen showing the script executing to start the splunk daemon (i.e. full of text), or "Only Start Splunk" A terminal window with just the BASH prompt and a terminal window with just the BASH prompt and no screen showing the script executing to start the splunk daemon (i.e. full of text). Either of the above cases means that the Splunk startup process had FAILED, and if you try to manually connect to the Splunk Web Server component in your browser (via the 127.0.0.1:8000 localhost URL and port for the Splunk Web Server) you get "unable to connect". Similarly, If you try to shutdown Splunk properly, by launching SLH again and selecting "Shudown" SLH does not properly execute the BASH script to shutdown Splunk properly (sigh). SOMETIMES Splunk will startup properly on the first try... but it is "iffy". I have NOT seen it properpy execute the "shutdown". IN ADDITION, it appears that if you put your Mac to sleep for an extended period (how long extended is I have not determined) that upon "resume" Splunk has gone into "hoo-hoo" land (a technical term). Arrggghhhh!! :geek on - with suppression of full "geek mode" in effect "WASSUP with THAT S***" you ask? Like many applications today Splunk is a "stack" of multiple component modules that must start in the proper order for the complete application to operate correctly. Splunk is written as a "core" server and a "web server" component to run the GUI (thus preventing the need to write and support multiple "native OS" GUIs. Simply put: SLH ain't gettin the job done. There is some issue in starting the components and properly waiting to start the next component in the stack (what is known as a "race condition"). When the proper sequence, with appropriate "wait time outs" for the components to start BEFORE the next component attempts to load, fails to occur the script errors out and SLH is left hanging. I have not done enough additional testing to determine "WTH" is going on, but I will try to gather some additional info (in my "spare time") and report the defect to the Splunkers for (hopefully!) a quick fix. :geek off - return to normal human mode "OK - that was "geek 2 me" (and I could care less) and how the hell do I get around the issue(s)" you ask?? SO... The key is to monitor the startup and shutdown processes and IF our good buddy SLH let's us down, take manual action to address the situation, as follows: 1) On startup AND shutdown Splunk opens a BASH shell in the MAC "Terminal" app and executes a script to start/stop the Splunk daemon (server and app stack). When it does that the MAC terminal application will open and the script(s) should run and display a "bunch of stuff". IF the terminal window opens and displays ONLY the BASH shell prompt Last login: Thu Oct 29 07:33:46 on ttys000 Daniels-MBP:~ page$ THEN - "Houston - we have a problem" IF this occurs, execute the following steps: 1) Select the "open apple" icon in the extreme upper left had corner of your screen (the app bar). 2) Select "Force Quit" 3) Select the "Splunk - not responding" process and force it to quit (this executes a LINUX kill command on the SLH process) 4) Answer the prompts to execute the kill command on SLH (don't worry about the first one - SLH has no "data" and answer the second with "ignore" since Apple does not need to get logs for this..) 5) Execute the SLH again and select "Start Only" and this time the MAC terminal window will show the BASH startup/shutdown script "bunch of stuff" which indicates the script is running properly. 6) This time SLH will display the appropriate "Splunk has started / stopped" message. 7) Now you can open your browser and enter "127.0.0.1:8000" and hit the enter key and you should see the Splunk GUI. BTW: The MAC version, in my experience, does NOT launch the GUI automatically, even when it opens the browser window and the manual connection to the Splunk Web Server via the "127.0.0.1:8000" is always required. After the first time you use the address/port, IF you do not do anything else which uses that the localhost IP address, it will default to the Splunk GUI by simply entering 127.0.0.1 and hitting enter. IF you get the "cannot connect to the server" message after your MAC has been in sleep mode, THEN you need to execute the above steps to properly shutdown Splunk AND THEN start splunk again, probably using the same steps for the startup process. Occasionally Splunk will start up correctly on the first try... but as stated above it seems to be "iffy". Hope this helps some folks. The work around described above has been tested repeatedly, by both Parin and myself, and works every time. Ain't computers wonderful! :-)) Regards, Dan ... View more