SplunkIt stuck @ ServerMain: Setting up agent(s)

[orgnalwood]

Anybody know why SplunkIt 2.02 would get stuck @ ServerMain: Setting up agent(s) under Splunk 6.x?

[guitar2002]

I have same issue on SplunkIt3.2 , Does anyone fixed it?

[gregopenit]

I was in the same case than you respecting the requirements of Splunkit v3.2 with:
- Firefox 17.0.10 on client-side
- xvfb 1.1.1-48.107 on client-side
- Python 2.6.8 on client-side and 2.6.6 on server-side
- Splunk 6.1.0 206881 on server-side

I was able to correct this freeze at "Setting up agent(s)" step by using another version of Splunk on server-side.
With Splunk-6.0.13-278006, everything works fine !

Client log :
[2017-03-01 11:44:28,473] ClientMain: Starting Pyro client with standalone=False, detach=False, and keep_alive=False
[2017-03-01 11:44:28,473] ClientMain: Initializing PyroClient object
[2017-03-01 11:44:28,473] ClientMain: Attempting to connect to server
[2017-03-01 11:44:28,492] ClientMain: Fetching server properties
[2017-03-01 11:44:28,545] ClientMain: Starting new agent thread
[2017-03-01 11:44:28,545] AgentControl: Initializing agent thread 0
[2017-03-01 11:44:28,546] AgentControl: Connected to server, received id d2c0t
[2017-03-01 11:44:28,546] AgentControl: Starting agent helper thread for agent thread 0
[2017-03-01 11:44:28,546] AgentControl: Received waiting status from agent helper thread 0
[2017-03-01 11:46:24,470] AgentControl: Setting up agent thread 0
[2017-03-01 11:46:24,571] AgentControl: Received setting up status from agent helper thread 0

Server log:
[2017-03-01 11:43:53,067] ServerMain: Preparing server and test driver object
[2017-03-01 11:43:53,067] TestDriver: Preparing Pyro test driver object
[2017-03-01 11:43:53,067] TestDriver: Preparing results logger with CSV output enabled
[2017-03-01 11:43:53,067] TestDriver: CSV fields: ['timestamp', 'unix_timestamp', 'test_name', 'test_class', 'host', 'thread', 'id', 'agent_run', 'client_run', 'elapsed', 'run_duration', 'username', 'event_count', 'result_count', 'time_to_first_event', 'time_to_search']
[2017-03-01 11:43:53,068] ServerMain: Starting pyro server thread
[2017-03-01 11:43:53,074] ServerMain: Starting watchdog thread
[2017-03-01 11:43:53,074] ServerMain: Starting in automatic mode
[2017-03-01 11:43:53,074] ServerMain: Waiting for 1 agent(s) to connect
[2017-03-01 11:44:28,546] TestDriver: User admin connected with id d2c0t from host splunktest.lan
[2017-03-01 11:44:28,546] ServerMain: 1 agent(s) connected!
[2017-03-01 11:44:28,546] SearchTestSetup: Reading scheduled_searches.txt for scheduled search strings, frequencies, and start/end times
[2017-03-01 11:44:28,569] SearchTestSetup: Performing setup for the SplunkIt Search Test
[2017-03-01 11:44:28,582] SearchTestSetup: Starting splunk, if not started already
[2017-03-01 11:44:28,681] SearchTestSetup: Removing file monitors from previous runs, if applicable
[2017-03-01 11:44:28,942] SearchTestSetup: Saving time range to bin/searchRecord.log, to be used later
[2017-03-01 11:44:28,959] SearchTestSetup: Adding splunkit_searchtest index, OR, clean the splunkit_searchtest index if it already exists
[2017-03-01 11:44:50,143] SearchTestSetup: Removing old saved seaches, including scheduled saved searches, from the server
[2017-03-01 11:44:56,303] SearchTestSetup: Detected old scheduled searches from an incomplete search test run. Cleaning up old scheduled searches before proceeding
[2017-03-01 11:45:24,305] SearchTestSetup: Creating new saved searches
[2017-03-01 11:45:59,795] SearchTestSetup: Adding necessary conf file entries to prepare the Splunk instance for the Search test
[2017-03-01 11:45:59,799] SearchTestSetup: Adding dashboards to server
[2017-03-01 11:45:59,800] SearchTestSetup: Copying splunkit_results.xml dashboard to server
[2017-03-01 11:45:59,812] SearchTestSetup: Copying splunkit_indexing_details.xml dashboard to server
[2017-03-01 11:45:59,818] SearchTestSetup: Restarting splunk
[2017-03-01 11:46:24,057] SearchTestSetup: Splunk restarted successfully
[2017-03-01 11:46:24,058] SearchTestSetup: Adding file monitors for streaming data and final results (when available)
[2017-03-01 11:46:24,422] TestDriver: Issuing setup command to all connected agents
[2017-03-01 11:46:24,423] ServerMain: Setting up agent(s)

[satishsdange]

could you please share some more information about your problem e.g. logs from splunkd

[orgnalwood]

nothing in the splunkd.log has any info, but here are the logs from spunkit-server & splunkit-client:
splunkit-server searchtest.log:
...
[2015-03-17 17:34:30,496] TestDriver: Issuing setup command to all connected agents
[2015-03-17 17:34:30,496] ServerMain: Setting up agent(s)
[2015-03-17 17:34:31,093] TestDriver: User admin (KZPso) disconnected
splunkit-user client.log:
[2015-03-17 17:32:54,039] ClientMain: Starting Pyro client with standalone=False, detach=False, and keep_alive=False
...
[2015-03-17 17:32:56,627] AgentControl: Initializing agent thread 0
[2015-03-17 17:32:56,635] AgentControl: Connected to server, received id KZPso
[2015-03-17 17:32:56,637] AgentControl: Starting agent helper thread for agent thread 0
[2015-03-17 17:32:56,638] AgentControl: Received waiting status from agent helper thread 0
[2015-03-17 17:34:28,884] AgentControl: Setting up agent thread 0
[2015-03-17 17:34:28,984] AgentControl: Received setting up status from agent helper thread 0
[2015-03-17 17:34:29,391] AgentControl: Agent helper thread 0 stopped unexpectedly. Stopping agent.
[2015-03-17 17:34:29,391] AgentControl: Shutting down agent thread 0
[2015-03-17 17:34:29,450] ClientMain: All agent threads stopped. Exiting client
[2015-03-17 17:34:29,450] ClientMain: Client done
[2015-03-17 17:34:29,450] ClientMain: Client exiting normally

[orgnalwood]

splunit-user stdout:

python bin/searchtest.py
server host: 10.154.134.21
server port: 8080
[2015-03-17 17:30:20,923] ClientMain: Starting Pyro client with standalone=False
, detach=False, and keep_alive=False
[2015-03-17 17:30:20,924] ClientMain: Initializing PyroClient object
[2015-03-17 17:30:20,924] ClientMain: Attempting to connect to server
[2015-03-17 17:30:25,436] ClientMain: Fetching server properties
[2015-03-17 17:30:25,493] ClientMain: Starting new agent thread
[2015-03-17 17:30:25,494] AgentControl: Initializing agent thread 0
[2015-03-17 17:30:25,505] AgentControl: Connected to server, received id dSyiv
[2015-03-17 17:30:25,505] AgentControl: Starting agent helper thread for agent thread 0
[2015-03-17 17:30:25,506] AgentControl: Received waiting status from agent helper thread 0
[2015-03-17 17:31:57,447] AgentControl: Setting up agent thread 0
[2015-03-17 17:31:57,548] AgentControl: Received setting up status from agent helper thread 0

[orgnalwood]

Exception in thread Thread-2:
Traceback (most recent call last):
File "D:\Python27\lib\threading.py", line 810, in bootstrap_inner
self.run()
File "D:\Users\tbanks-admin\Downloads\splunkit_32\splunkit\splunkit-user\splun
kit-user\lib\pyro_client.py", line 564, in run
self.agent_instance = agent_test_class(properties=self.properties)
File "D:\Users\tbanks-admin\Downloads\splunkit_32\splunkit\splunkit-user\splun
kit-user\lib\pyro_test_classes.py", line 27, in __init
password = self.properties['password']
File "D:\Users\tbanks-admin\Downloads\splunkit_32\splunkit\splunkit-user\splun
kit-user\lib\kirsch.py", line 34, in init
self.driver = webdriver.Firefox(self.profile)
File "D:\Python27\lib\site-packages\selenium-2.45.0-py2.7.egg\selenium\webdriv
er\firefox\webdriver.py", line 59, in init
self.binary, timeout),
File "D:\Python27\lib\site-packages\selenium-2.45.0-py2.7.egg\selenium\webdriv
er\firefox\extension_connection.py", line 47, in init
self.binary.launch_browser(self.profile)
File "D:\Python27\lib\site-packages\selenium-2.45.0-py2.7.egg\selenium\webdriv
er\firefox\firefox_binary.py", line 65, in launch_browser
self._start_from_profile_path(self.profile.path)
File "D:\Python27\lib\site-packages\selenium-2.45.0-py2.7.egg\selenium\webdriv
er\firefox\firefox_binary.py", line 88, in _start_from_profile_path
env=self._firefox_env).communicate()

[orgnalwood]

File "D:\Python27\lib\subprocess.py", line 710, in init
errread, errwrite)
File "D:\Python27\lib\subprocess.py", line 958, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified

[orgnalwood]

splunkd_stderr.log (showing when the searchtest script restarts splunk):
2015-03-17 17:30:27.702 -0700 Interrupt signal received
2015-03-17 17:30:38.808 -0700 splunkd started (build 245427)
2015-03-17 17:31:41.716 -0700 Interrupt signal received
2015-03-17 17:31:57.747 -0700 splunkd started (build 245427)
2015-03-17 17:32:58.832 -0700 Interrupt signal received
2015-03-17 17:33:09.950 -0700 splunkd started (build 245427)
2015-03-17 17:34:13.093 -0700 Interrupt signal received
2015-03-17 17:34:29.116 -0700 splunkd started (build 245427)