Join the Conversation
- Find Answers
- :
- Splunk Administration
- :
- Monitoring Splunk
- :
- splunkweb reporting splunkd timeout
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
I am currently running into problems with my Search Heads. Users are experiencing intermittent timeouts of splunkd, which is stated on by Splunkweb during search, log in, etc. When Splunk is started using the CLI I receive the message “timed out waiting for splunkd”, but splunkweb starts up immediately. Search Head configuration has not changed in months. Splunkd log does not report any errors and warnings when messages are presented by splunkweb.
Configurations:
- Search Head pooling
- Splunk 5.0.5
- Windows Server Standard 2008R2
- Virtualized
Suggestions?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
During my investigation I noticed that Splunk was taking an exuberant amount of time to read its config file. This come to light when I started running processmon. My initial thought was ACL or network issues. After watching read/write actions with processmon it appeared that I had intermittent disk transfer (IOP) issues with my cif.
To confirm that I switch search head pooling to disabled. Once disabled Splunk web function as expected. My next step was to create an nfs share on a remote system and copying all pool configurations to the new nfs share. Splunkweb worked like a charm. I switch back and forth between local, windows nfs share, net app share, and the cif share. Not very scientific, but worked. To get my infrastructure teams to even look at the problem I need concrete numbers, so I ran ActiveIO tests against all four setups.
Results (spread sheet too big):
- Local disk fast (as expected)
- windows nfs (slower but acceptable)
- Net App share (on par with windows nfs)
- cif share was slower by a factor of 15 (unusable)
IO Zone command for full test (one thread):
- iozone -a –b <drive_for_output>:\output.xls -f <drive_to_test>:\iozone.tmp –Q
IO Zone command for quick test (four threads):
- iozone -i 0 -t 4 -F <drive_to_test>:\iozone.tmp <drive_to_test>:\1.tmp <drive_to_test>:\2.tmp <drive_to_test>:\3.tmp -Q
Another Helpful doc was pooling configuration issues. For those of you who looked at this thanks.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
During my investigation I noticed that Splunk was taking an exuberant amount of time to read its config file. This come to light when I started running processmon. My initial thought was ACL or network issues. After watching read/write actions with processmon it appeared that I had intermittent disk transfer (IOP) issues with my cif.
To confirm that I switch search head pooling to disabled. Once disabled Splunk web function as expected. My next step was to create an nfs share on a remote system and copying all pool configurations to the new nfs share. Splunkweb worked like a charm. I switch back and forth between local, windows nfs share, net app share, and the cif share. Not very scientific, but worked. To get my infrastructure teams to even look at the problem I need concrete numbers, so I ran ActiveIO tests against all four setups.
Results (spread sheet too big):
- Local disk fast (as expected)
- windows nfs (slower but acceptable)
- Net App share (on par with windows nfs)
- cif share was slower by a factor of 15 (unusable)
IO Zone command for full test (one thread):
- iozone -a –b <drive_for_output>:\output.xls -f <drive_to_test>:\iozone.tmp –Q
IO Zone command for quick test (four threads):
- iozone -i 0 -t 4 -F <drive_to_test>:\iozone.tmp <drive_to_test>:\1.tmp <drive_to_test>:\2.tmp <drive_to_test>:\3.tmp -Q
Another Helpful doc was pooling configuration issues. For those of you who looked at this thanks.
Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!
Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.
Event Series: The Agentic SOC: Trust Before Autonomy
Free Professional Services for .conf26 Attendees
Defend at Machine Speed: Your Guide to Security Sessions at .conf26
