One reason is restarting splunkd. You can configure your retention policy based on time instead of size. ... View more

Check out the inventory of REST APIs they may have what you are looking for Example: http://docs.splunk.com/Documentation/Splunk/6.4.2/RESTREF/RESTapps ... View more

Not sure i understand what you are saying "fields contain xml lines". Have you tried using this custom command ".... | xmlmv" ... View more

Can you explain where does the single event start and stop? What are the logical boundaries around an event? I see timestamp in every line (which in normal logs are single events) but they dont appears to be the case here. ... View more

Thank you for confirming that! ... View more

everything is splunk is configurable (including the UI). I have done it in the past but i cannot remember the instructions. Search this forum for customizing the interface. It will boil down to custom xmls. ... View more

In the latest release of Hunk 6.4 there is a support for Data Model Acceleration and all the commands that goes with Data Models http://docs.splunk.com/Documentation/Hunk/6.4.0/Hunk/Configuredatamodelacceleration ... View more

Here is another one @(?\w+.\w{3}) ... View more

Simple answer, yes it works. As for Indexer discovery question, this has to do with the HF (in your case) discovering any new IDX added to the cluster. The UF or HF are no longer required to be configured (outputs.conf) ahead of time. This feature only works for clustered indexers. ... View more

I believe splunk uses standard zlib library. There is py script you can playing with but I strongly recommend against it. My guess they set the compression ratio to strike a balance between speed vs space. root:/var/root # locate zlib|grep splunk /opt/splunk/6.3/lib/python2.7/encodings/zlib_codec.py /opt/splunk/6.3/lib/python2.7/lib-dynload/zlib.so ... View more

The docs say a 100G incoming is broken to 15% for raw data (journal.gz file) and 35% for meta data (tsidx files). So your 100G will occupy ~50G space. Be aware that is an average. Different ASCII files has various compression ratios (base on repeated patterns). ... View more

Here is a simple script that test for port 8089 connectivity. It will send email alert (and restart the process if not connection): /export/home/mhassan/scripts $ cat splunkd-watchdog.sh !/bin/bash splunkd-watchdog.sh Monitor Splunk port 8089. Restart process if no answer -Mohamad Hassan 2/2/215 timestamp= date USER= whoami WHO= who -u host= uname -n LOGFILE="/opt/splunk/var/log/splunk-watchdog.log" PROC="splunkd" PORT="8089" ADMINS="[email protected] [email protected]" TEST= nc -z -w5 $HOSTNAME $PORT;echo $? < /dev/null Debugging let TEST=0 echo $TEST if [ $TEST -eq 1 ] ;then /opt/splunk/bin/splunk restart RESULT=`ps ax | grep -i "$PROC" | grep -v grep|grep -v watchdog` echo "$timestamp $HOSTNAME: splunk-watchdog ALERT! $PROC process restarted" >> $LOGFILE echo -e "Splunk watchdog ALERT! Splunk port $PORT is not answering.\n\n Hostname: $HOSTNAME\n Time: $timestamp\n Notifying: $ADMINS\n ACTION: Restarting ($PROC)!\n\nCurrrent logged-in users:\n$WHO\n\nCurrent splunkd status(after restart):\n$RESULT" |mail -s "$PROC watchdog ALERT! [$HOSTNAME]" $ADMINS echo "$PROC restarted!" echo "Email sent to: $ADMINS" fi ... View more

Please be aware if you are in a search pool environment (using NFS for example) you need to copy framework directory to the centralized area. Example: cp /opt/splunk/etc/apps/framework nfs:/sharedmount/etc/apps chmod -R 755 /shareemount/etc/apps/framework ... View more

This is true for the "upgrade" process (using RPM or gz files). It is not true if you are using deployment server to push new apps. The old directory (xyz-app directory) always get deleted and push from the deployment server. I don't know of better way of doing this! ... View more

try custom log configuration on your WSG produce text log files. Then use syslog-ng v3 or Snare agent to bounce the logs to splunk (or centeral logging host) ... View more