We currently have multiple search heads with a load balancer in front of them. Users are logged out if a search head goes down. Requesting that sessions are replicated between search head and one peer search head so that searches will continue(in the background) and session can continue if a search head goes down or is restarted. Thanks, Rob ... View more

Trying to determine the 800-1200+ IOPS requirement is being met based on http://wiki.splunk.com/Community:HardwareTuningFactors I'm not able to install Bonnie++ due to company restrictions. How can I calcualte IOPS? Runing on RedHat Linux 6 (2.6.18-238.9.1.el5 #1 SMP Fri Mar 18 12:42:39 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux) I have some data below but not sure how to translate it into IOPS. sar from sysadmin Time: 08:56:29 PM Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sdb 168.69 0.53 3.30 9477500 58565203 Time: 08:56:39 PM Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sdb 1110.40 8.93 14.69 89 146 Time: 08:56:49 PM Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sdb 721.60 10.49 5.63 104 56 Time: 08:56:59 PM Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sdb 378.52 2.23 4.89 22 48 Time: 08:57:09 PM Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sdb 528.50 10.42 8.09 104 80 Time: 08:57:19 PM Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sdb 1861.58 7.37 33.73 73 337 Time: 08:57:29 PM Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sdb 1550.80 11.69 34.63 116 346 iostat from *nix app Device rReq_PS wReq_PS rKB_PS wKB_PS avgWaitMillis avgSvcMillis bandwUtilPct sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb 0.00 675.25 0.00 10126.73 54.19 0.49 33.17 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-5 0.00 3685.15 0.00 14740.59 39.33 0.09 33.17 ... View more

It's quite easy to report in splunk on what type of events you have but how to report on what types of events you don't have? What I know how to do: Summarize results by host and soucetype where there are 1 or more events. What I don't know how to do: Summarize results by host and sourcetype where there are 0 events. Example search which works for event matches but doesn't include zero matches: host=hosta OR host=hostb OR host=hostc|stats count(host) by sourcetype host Is there a way I can return a zero count result for a host or sourcetype in splunk which doesn't have events? sourctype host count(host) access hosta 500 access hostb 250 access hostc 0 NOTE: I will be running for over 1,000 hosts. Some of these hosts have never sent data to splunk. This is quite important when you want to audit an environment to ensure you are collecting events. Thanks, Rob ... View more

I've encountered the following with a crashed splunk forwarder running on 4.3.3 Linux 64-bit. Splunk says it’s running…. $ ./splunk status splunkd is running (PID: 14371). splunk helpers are running (PIDs: 14372). But a ps –ef shows different No, splunk is not running… $ ps -ef|grep splunk|grep -v grep splunk 6569 6560 0 18:04 pts/2 00:00:00 -ksh splunk 26616 6569 0 18:19 pts/2 00:00:00 ps –ef Seems different processes have taken over these pids… $ ps -ef|egrep "14371|14372" root 14371 2387 0 Jul23 ? 00:00:00 [rpciod/27] root 14372 2387 0 Jul23 ? 00:00:00 [rpciod/28] Splunk is trying to stop the wrong pid!… $ ./splunk stop Stopping splunkd... Shutting down. Please wait, as this may take a few minutes. Could not kill pid 14371. [FAILED] The start command also thinks splunk is running but it is not… $ ./splunk start The splunk daemon (splunkd) is already running. [FAILED] I remove the stale pid file… rm splunkforwarder/var/run/splunk/splunkd.pid Splunk now starts… $ ./splunk start Splunk> Like an F-18, bro. Checking prerequisites... Checking mgmt port [8089]: open Checking conf files for typos... All preliminary checks passed. Starting splunk server daemon (splunkd)... [ OK ] I've found the following bug metioned in the 4.3.4 documentation. http://docs.splunk.com/Documentation/Splunk/4.3.4/ReleaseNotes/KnownIssues Forwarder startup script should handle stale PID files gracefully after server crashes. (SPL-36597) Is this an issue in 4.3.4 as well or corrected in 4.3.4? Also, I would like to have the splunk status check that the splunk proceses are running, not just that a pid is running. Thanks, Rob ... View more