So, on the search heads, does $SPLUNK_HOME/etc/apps/props_conf/local look fine? I would check $SPLUNK_HOME/bin/splunk btool --app=props_conf props list on the search head and compare it with the single server's outout of the same command. ... View more

Oh Ok, thanks for the clarification. Now thanks to the input I know how to display index time and the event time. Thanks again. ... View more

I forgot to mention that it's Splunk Enterprise 6.5.6 ... View more

See my answer to this problem here: http://answers.splunk.com/answers/104772/how-to-restart-just-a-cluster-master/114460 It doesn't eliminate the problem, but helps the Cluster Master recover faster. ... View more

Upgrade steps in short: These steps are for 1 SH, 1 CM, 1 FWD, 4 IDX 1) set CM in maintenance mode 2) bring down all the peers 3) bring down the master 4) upgrade everybody - don't start! (untar over existing /splunk directory) 5) start master 6) set master in maintenance mode 7) start peers 8) start search-head 9) check things (searchability from SH, show cluster-bundle-status from CM) 10) if all is well, unset maintenance mode, then start the fwd 11) test fwd / inputs 12) pantaloons 13) doubloons ... View more

Clustering is not expected to use 20GB of RAM. In our tests we have not seen it go over 3GB. So, I would think that there is a associated memory leak. Which version of splunk is it? Please file a support case and we will help find the leak. ... View more

I am going to glom onto this post and try to get better details then "depends" to the question. I understand that all outliers depend.. but lets keep this simple. *NIX app How do you do deployment across a cluster designed NSPOF: 4 x Indexer 1 x Cluster mater 1 x Deployer / license server 3 x Search nodes in SH Cluster (VIP for HA) 2 x Heavy forwarders collecting syslog /snmp / json (VIP for HA on inbound) I did read Documentation: http://docs.splunk.com/Documentation/Splunk/6.2.5/DistSearch/HowconfigurationworksinSHC Design for file location: splunkcmaster01 -> /opt/splunk/etc/masterapps/ (used for index node component deployment) splunkdeploy01 -> /opt/splunk/etc/shcluster/apps/ (used for Search Head Cluster Nodes) -> /opt/splunk/etc/deployment.apps/ (Forwarders) Example: *NIX app deployment (via best practice) Step 1: Download splunk-app-for-unix-and-linux_503.tgz onto splunkdeploy01 Step 2: copy files from within tarball out to different splunk directories [root@splunkdeploy01 /]# cd /tmp/ [root@splunkdeploy01 tmp]# cp /media/labfiles/Software/Splunk/apps/splunk-app-for-unix-and-linux_503.tgz . [root@splunkdeploy01 tmp]# tar -zxvf splunk-app-for-unix-and-linux_503.tgz [root@splunkdeploy01 tmp]# cd splunk_app_for_nix/ ?? < missing explaination of what dir / files goes where. I think /SA-nix goes in one folder and /TA_nix goes in another folder but also some parts go onto search head. > ?? [root@splunkdeploy01 /]# ls /tmp/splunk_app_for_nix/install/ SA-nix Splunk_TA_nix Question: 1) README.txt within app has no info to guide on this procedure. And docs website does not speak to cluster type deployment. Is there a guide for or youtube for how this is done? Once I get the answer for this I can post this.. and as I crawl through other apps deploy on cluster.. I can post that. (example DB connect , Exchange, VWare etc.. which say a lot of "deploy like standalone" but peices to the process are missing... or I am missing the "Very F*ine* Manual" ... View more

maybe check my answer here (considering this page is the top hit in google): http://answers.splunk.com/answers/151938/why-is-cluster-master-stuck-at-bundle-validation-is-in-progress-indefinitely-after-configuration-bundle-update.html#answer-172463 ... View more

Are you trying to send 5000 Bytes over UDP with syslog? If you use syslog I think that it could be a maximun Byte limit on the protocol. ... View more

Any update? ... View more

I would suggest using a sourcetype in place of source. Since the source will be set to each file under the directory being monitored, the transforms may not match. You can try the following: On fwd: [monitor:///home/tomcat1/apache-tomcat-6.0.24/logs] disabled=false host=prod_228_1 index=production sourcetype=apache_logs On indexer: [sourcetype::apache_logs] TRANSFORMS-null=setnull [setnull] REGEX = healthCheck DEST_KEY = queue FORMAT = nullQueue ... View more

Agreed - Its the quantity of files and the directory structure which makes more of a difference. You can have 50GB of data in 10 files and it'll work fine, if you then have 50GB of data in 5000 files across 50 directories your going to see CPU going high. ... View more