The mentions of an "error" are benign and can be safely ignored. This cosmetic issue is well-known and has been filed under reference SPL-34584. ... View more

If you want to track the size of the archive processing queue, you could run the following search : index=_internal host=host_reading_archives source=*metrics.log group=queue name=aq earliest=-24h | timechart span=15mn perc95(current_size), max(max_size) Feel free to change the values of "earliest" and "span" to better fulfill your needs. Note that we are asking for the 95th percentile of the "current_size" field and the maximum value of the "max_size" field to produce a good statistical representation of what's going on. As for the remediation to the blockage of the archive processing queue, the answer and comments from Stephen Sorkin in the post you refer to are still valid. I would add a few : If at all possible, feed the files to Splunk when uncompressed. Uncompressed files can be processed in parallel where archives have to be processed serially. Find out if there other blocked queues downstream which might be the actual culprit by propagating the clogging upstream. You can do this by running a search like index=_internal host=host_reading_archives source=*metrics.log group=queue | timechart span=15mn perc95(current_size) by name . Change span or the value of "host" as needed, depending on where (forwarder reading the archives? indexer committing the events to disk?) and when you want to look at the size of the event processing queues. Optimize event-processing for the sourcetypes indexed from the archive files by declaring explicit line-breaking rules (use LINE_BREAKER in props.conf) and time-stamp extractions (use TIME_FORMAT, TIME_PREFIX and MAX_TIMESTAMP_LOOKAHEAD in props.conf). See props.conf.spec - http://www.splunk.com/base/Documentation/latest/Admin/Propsconf - for details. If your architecture allows it, split the task of reading archive files between several Splunk forwarders. ... View more

The following configuration procedure has been written precisely to address this case : http://www.splunk.com/wiki/Community:Splunk2Splunk_SSL_3rdPartyCA If you are unable to configure SSL for your splunk2splunk communication with these instructions, please attempt to follow the troubleshooting steps on that page (section #5) and paste here what you can from the btool output for inputs/outputs.conf and the pertinent (TcpInputProc/TcpOutputProc) splunkd.log lines. ... View more

There is currently no way to configure a scheduled search not to run or generate alerts during a specific time frame. If you're lucky, the blackout period you want to introduce is easy enough to bake into the cron format of the "cron_schedule". This is however not the case with the example discussed here. In our case, we will have to play with the date_hour and date_minute internal fields to exclude events that would normally match the search. We want only events that match the following criteria : Can be found in source /var/log/messages : source=/var/log/messages Contain the string "shutdown" or the string "reboot" : ("shutdown" OR "reboot") Did not happen at 12AM (date_hour=0), after the 15th minute (date_minute>14) and before the 45th minute (date_minute<46) of that hour : NOT (date\_hour=0 AND date\_minute>14 AND date_minute<46) The following search string will achieve that goal : source=/var/log/messages ("shutdown" OR "reboot") NOT (date_hour=0 AND date_minute>14 AND date_minute<46) | stats count by host ... View more

To my knowledge, there are no showstoppers or blocking issues with a volume manager such as LVM or Veritas Volume Manager being in charge of the filesystem that hosts your Splunk indexes. The things to be careful with would be the same than with any other I/O intensive application. For example, if your volume management software allows dynamic volume and filesystem growth but this operation has a potentially heavy impact on your I/O performance, you might want to stop Splunk while you dynamically expand your storage capacity so as to not mess with indexing. The same could be said for live snapshots, again in the case that they would be disruptive to your I/O performance. That being said, it remains important to adequately manage the underlying topology of the logical devices assigned to Splunk's indexes in order to maintain optimal performance. As a reminder, the ideal topology for Splunk indexes is hardware RAID10 : http://www.splunk.com/base/Documentation/latest/Installation/CapacityplanningforalargerSplunkdeployment From a support perspective, although volume management software solutions are not part of our Quality Assurance cycle, they are supported to host Splunk's indexes as long as they are used in a reasonable manner and the indexes are hosted on supported filesystems. ... View more

Checking the line count of a source file against the number of lines indexed by Splunk can be easily achieved. Here is an example with a file that numbers 1117 lines indexed as 7 events : Count the number of lines in the source file excluding all lines empty or exclusively containing white space characters, which Splunk doesn't index : [root@beefysup01 ~]# grep -v -e "^\s*$" /var/log/Xorg.0.log | wc -l 1116 In Splunk, search for all events for that specific source over all time, and aggregate the values of the "linecount" field : source="/var/log/Xorg.0.log" | stats sum(linecount) The two numbers should match, provided that you do not work on a live file that is part of a rotation (example : /var/log/messages or $SPLUNK_HOME/var/log/splunk/metrics.log) or that you are routing events from this file to the null queue. Another method, although often less accurate, is to measure the byte count of the source file (again, excluding empty lines) and compare it against the aggregated byte count for all events indexed for that source : Count the number of bytes in the source file excluding all lines empty or exclusively containing white space characters, which Splunk doesn't index : [root@beefysup01 ~]# grep -v -e "^\s*$" /var/log/anaconda.log | wc -c 841067 In Splunk, search for all events for that specific source over all time, calculate and aggregate the size of all events in the "esize" field using the len() eval function on the event raw data (the "_raw" field) and add the event count for the file!. The last step is important to get an accurate byte count because Splunk "loses" one byte per event when it dumps the last newline character of each event : source="/var/log/anaconda.log" | eval esize=len(_raw) | stats sum(esize) AS sum_esize, count | eval fsize=sum_esize + count | fields fsize ... View more

Your CHARSET and NO_BINARY_CHECK settings seem appropriate. I would worry that the source-based props.conf spec might not always match, though. I would advise to assign a sourcetype to this type of logs at the input level if you haven't done so yet ("sourcetype=sqlerror" in the corresponding inputs.conf stanza) and to set up a sourcetype-based stanza in props.conf to replace the one you currently have : [sqlerror] CHARSET = UTF-16LE NO_BINARY_CHECK = true This will be more reliable than a source-based spec. ... View more

This is indeed a new 4.2 functionality. From the CLI, one can simply run : $SPLUNK_HOME/bin/splunk add <index_name> $SPLUNK_HOME/bin/splunk reload index The newly-created index will be ready to use at this point. The UI will do the "splunk reload index" for you, so any indexes created from the UI are ready to be used right away. One caveat here is that on a system where splunkd is very busy, there might be a delay between the execution of the index reload and the actual availability of the index to throw data at. As a rule of thumb, this delay is usually measured in seconds, but if you want to be sure that the index will be available, I would recommend to wait for 30 seconds to 1 minute after the dynamic reload before sending data to it. Do note that "splunk reload index" only allows to add new indexes. You will not be able to remove a pre-existing index on the fly in this way. ... View more

This error means that the search artifact (the file package containing the search results) requested by Splunk Web could not be found in $SPLUNK_HOME/var/run/splunk/dispatch for the search that was just dispatched. In the error show above, the path to the real-time search artifact requested should be $SPLUNK_HOME/var/run/splunk/dispatch/rt_1302277839.33, but there is nothing there. This problem commonly happens when the $SPLUNK_HOME/var/run/dispatch directory is hosted on a network device with a time setting behind the system clock of the operating system where splunkd is running. What happens then is that the search artifacts are created in the dispatch directory with a modification time behind the system time known to splunkd : [root@splunk-search-head dispatch]# pwd /opt/splunk/var/run/splunk/dispatch [root@splunk-search-head dispatch]# touch test ; ls -l test ; date -rw-r--r-- 1 root root 0 Apr 22 22:16 test <=== creation time assigned to the file by the NAS device : 22:16pm Fri Apr 22 22:39:58 GMT 2011 <=== system time at which the file was created : 22:39pm As you can see here, any file newly-created in the dispatch directory has a modification time 23 minutes behind the system time. This means that any new search artifact created by an ad-hoc search will be evaluated by splunkd as having exceeded the default TTL (Time To Live) of 15 minutes and will therefore be immediately deleted. For that reason, it is very important to ensure that the clock of any NAS device hosting the search dispatch directory is in synch with the clock of the system on which splunkd is running. ... View more

These errors are benign and are occurring because of an orphaned stanza for the legacy "splunklogger" index in $SPLUNK_HOME/etc/system/default/indexes.conf : # this index has been removed in the 4.1 series, but this stanza must be # preserved to avoid displaying errors for users that have tweaked the index's # size/etc parameters in local/indexes.conf. [splunklogger] disabled = true The goal of this stanza is to disable the "splunklogger" index which has been deprecated for a while. Splunk diag is a little bit anal about index stanzas that fail to include non-optional parameters such as "homePath", "coldPath" or "thawedPath" which is why you are seeing this error. The easiest way to suppress this error would be to modify the [splunklogger] stanza in $SPLUNK_HOME/etc/system/default/indexes.conf (or $SPLUNK_HOME/etc/system/local/indexes.conf if you want this to be upgrade-resilient) as follows : # this index has been removed in the 4.1 series, but this stanza must be # preserved to avoid displaying errors for users that have tweaked the index's # size/etc parameters in local/indexes.conf. [splunklogger] disabled = true homePath = $SPLUNK_DB/splunklogger/db coldPath = $SPLUNK_DB/splunklogger/colddb thawedPath = $SPLUNK_DB/splunklogger/thaweddb In all likelihood, this minor cosmetic error will be fixed in a future release. ... View more

As you may already know, on *nix systems the start-up of Splunk during system boot is normally controlled by the /etc/init.d/splunk script. Looking at the "start" procedure in that script, we use "--no-prompt" as the default start-up option : splunk_start() { echo Starting Splunk... "/home/support/sas/bin/splunk" start --no-prompt RETVAL=$? [ $RETVAL -eq 0 ] && touch /var/lock/subsys/splunk } This is in concordance with the behavior reported here when considering the definition of the "--no-prompt" option - http://www.splunk.com/base/Documentation/latest/Installation/StartSplunkforthefirsttime#Other_start_options : There are two other start options: no-prompt and answer-yes: * If you run $SPLUNK_HOME/bin/splunk start --no-prompt, Splunk proceeds with startup until it requires you to answer a question. Then, it displays the question, why it is quitting, and quits. * If you run SPLUNK_HOME/bin/splunk start --answer-yes, Splunk proceeds with startup and automatically answers "yes" to all yes/no questions. Splunk displays the question and answer as it continues. You could add "--answer-yes" after "--no-prompt" in the start procedure of /etc/init.d/splunk which would result in the following behavior on start-up : Splunk answers "yes" to any "yes/no" question. Splunk quits when it encounters a non-"yes/no" question. Now keep in mind that this is a change that you would have to make at your own risk. It might be preferable for the Splunk admin to be clearly aware of the existence of index consistency issues. But if you would like to change the behavior of /etc/init.d/splunk in the way we just discussed, that's how you can do it. IMPORTANT UPDATE : As of Splunk 4.2.3, this prompt no longer occurs and the user is notified instead upon restart after an unclean shutdown that a manual check of the indexes might be a good idea : Splunk has detected an unclean shutdown. The database should be checked in order to ensure correct search results, but this may take a very long time, depending on your system. If you would like to check/repair the database, stop Splunk and run: splunk fsck --all --repair ... View more