Here is a detailed procedure to use non-default (in this case, self-signed) SSL certificates with common-name-based authentication (for the indexer(s) only) in a splunk2splunk (indexer to forwarder) connection. 1 - On a Splunk instance, create your own root certificate : (As per http://www.splunk.com/base/Documentation/latest/Admin/SecureaccesstoyourSplunkserverwithSSL#Generate_a_new_root_certificate) NOTE : For clarity's sake, it is better to generate new certificates in another directory than $SPLUNK_HOME/etc/auth in order not to overwrite those that exist there. In our example, we will create and use $SPLUNK_HOME/etc/certs. This will also ensure that you can keep using the certificates that ship with Splunk in $SPLUNK_HOME/etc/auth for other Splunk components if you wish to do so. # mkdir $SPLUNK_HOME/etc/certs Point openssl to Splunk's openssl.cnf : # export OPENSSL_CONF=$SPLUNK_HOME/openssl/openssl.cnf Generate a new root certificate : # $SPLUNK_HOME/bin/genRootCA.sh -d $SPLUNK_HOME/etc/certs => This will create a new certificate authority public certificate in $SPLUNK_HOME/etc/certs/cacert.pem This public CA certificate is to be distributed to all Splunk instances (indexers and forwarders) who will be checking server certificates signed with the root certificate we just generated (ca.pem). 2 - Generate a new self-signed server certificate for your indexer, specifying your indexer's host name as the common name recorded in the certificate : (As per http://www.splunk.com/base/Documentation/latest/Admin/SecureaccesstoyourSplunkserverwithSSL#Generate_a_new_signed_certificate_and_private_key_pair) In our example, let's assume that your indexer's host name is "splunk-idx-01.example.com". Let's also assume that you want to use "changeme" as the password for your indexer's server certificate. # $SPLUNK_HOME/bin/genSignedServerCert.sh -d $SPLUNK_HOME/etc/certs -n splunk-idx-01 -c splunk-idx-01.example.com -p Enter "changeme" as the PEM pass phrase. Enter "splunk-idx-01.example.com" as the common name All other values are optional. => This will create a new server certificate in $SPLUNK_HOME/etc/certs/splunk-idx-01.pem. We will copy this server certificate to the indexer "splunk-idx-01.example.com, who will present it to the forwarders. The forwarders will use the public CA certificate from step 1 (cacert.pem) to check the identity of the indexer. 3 - Generate a new self-signed server certificate for your forwarders : In our example, let's assume that you want to use "changeme2" as the password for your forwarders' server certificates. # $SPLUNK_HOME/bin/genSignedServerCert.sh -d $SPLUNK_HOME/etc/certs -n forwarder -p Enter "changeme2" as the PEM pass phrase. All other values are optional. => This will create a new server certificate in $SPLUNK_HOME/etc/certs/forwarder.pem. We will propagate this server certificate to all of our forwarders, who will present it to the indexer(s) when a connection is established. The indexer(s) will use the public CA certificate we generated in step 1 (cacert.pem) to check the credentials of the forwarders. 4 - Set up the indexer to use the newly created server certificate and to check the forwarders' certificates : First, copy the $SPLUNK_HOME/etc/certs/splunk-idx-01.pem and $SPLUNK_HOME/etc/certs/cacert.pem files to your indexer and put them in a newly created $SPLUNK_HOME/etc/certs directory. We will assume here that you will be using port 9997 to receive data from your forwarders. In $SPLUNK_HOME/etc/system/local/inputs.conf, set up the following stanzas : [SSL] rootCA = $SPLUNK_HOME/etc/certs/cacert.pem serverCert = $SPLUNK_HOME/etc/certs/splunk-idx-01.pem password = changeme requireClientCert = false [splunktcp-ssl:9997] compressed = true Important note regarding requireClientCert : As of Splunk 4.2.4, setting requireClientCert = true in the indexer's inputs.conf will cause forwarding to fail! A bug (SPL-37637) is currently open to address this issue. In the meantime, keep requireClientCert set to "false". Setting "requireClientCert = true" would require the following conditions to be met : "rootCA" must point to a file containing the CA's public key, without a password. In our example, it's the cacert.pem file we generated in step 1. The forwarder's server certificate defined by "sslCertPath" in outputs.conf (see step 5) is signed by that CA. The forwarder has the password to read his own certificate ("sslPassword" in outputs.conf, as defined in step 5). This password is "changeme2" in our example. The purpose of this setup is to ensure that only forwarders that you have distributed a signed certificate to can connect to this indexer. Restart Splunk after making these changes. Note that this will hash the certificate password in inputs.conf. # $SPLUNK_HOME/bin/splunk restart 5 - Set up your forwarder(s) to use the newly created server certificates and to check the indexer's certificate with common name authentication : Copy the $SPLUNK_HOME/etc/certs/forwarder.pem and $SPLUNK_HOME/etc/certs/cacert.pem files to your forwarder(s) and put them in a newly created $SPLUNK_HOME/etc/certs directory. Then, define the following stanzas in $SPLUNK_HOME/etc/system/local/outputs.conf. Let's assume that your indexer's IP address is 192.168.1.100 : [tcpout] defaultGroup = splunkssl [tcpout:splunkssl] server = 192.168.1.100:9997 compressed = true [tcpout-server://192.168.1.100:9997] sslRootCAPath = $SPLUNK_HOME/etc/certs/cacert.pem sslCertPath = $SPLUNK_HOME/etc/certs/forwarder.pem sslPassword = changeme2 sslVerifyServerCert = true sslCommonNameToCheck = splunk-idx-01.example.com altCommonNameToCheck = splunk-idx-01 Note that we have set "sslVerifyServerCert = true". This requires the following conditions to be met : "sslRootCAPath" must point to a file containing the CA's public key, without a password. In our example, it's the cacert.pem file we generated in step 1. The indexer's "serverCert" (as defined in inputs.conf on step 4) is signed by that CA. The indexer has the password to read his own certificate ("password" in inputs.conf, as defined in step 4). This password is "changeme" in our example. The common name on the indexer's server certificate ("serverCert" in inputs.conf, as defined in step 4) matches one of the "sslCommonNameToCheck" settings. This setup ensures that the client/forwarder only forwards to someone with a certificate signed by the CA and that that certificate was issued to someone with the expected name, i.e., so that the client only sends to someone to whom that certificate is specific. The name doesn't have to match the DNS, hostname, servername, or IP or anything outside of these files. (Thus this is not subject to DNS SSL attacks). All that needs to match is the name in the Splunk configuration file and the name signed into the indexer's certificate. If you are in a distributed environment and therefore have one server certificate per indexer, set up one [tcpout-server:...] stanza per indexer with "sslServerCertPath" pointing to each individual server certificate. Finally, restart the forwarder : # $SPLUNK_HOME/bin/splunk restart 6 - You should now see the indexer and the forwarder establishing an SSL connection. Here's what it looks like in $SPLUNK_HOME/var/log/splunk/splunkd.log on both sides. on the indexer (with category.TcpInputProc=DEBUG in $SPLUNK_HOME/etc/log.cfg) : (During start up...) 09-21-2010 18:47:29.735 DEBUG TcpInputProc - Initializing 09-21-2010 18:47:29.735 INFO TcpInputProc - using queueSize 1000 09-21-2010 18:47:29.735 DEBUG TcpInputProc - creating tcp pipelineData queue 09-21-2010 18:47:29.735 DEBUG TcpInputProc - Reconfiguring 09-21-2010 18:47:29.735 DEBUG TcpInputProc - readConfig - clearing maps 09-21-2010 18:47:29.735 DEBUG TcpInputProc - global prop enables2sHeartbeat=true 09-21-2010 18:47:29.735 DEBUG TcpInputProc - global prop s2skeepaliveTimeout=600 09-21-2010 18:47:29.735 DEBUG TcpInputProc - global prop inputShutdownTimeout=90 09-21-2010 18:47:29.735 DEBUG TcpInputProc - readConfig - scanning configs 09-21-2010 18:47:29.735 DEBUG TcpInputProc - SSL serverCert=/opt/splunk/etc/certs/splunk-idx-01.pem 09-21-2010 18:47:29.735 DEBUG TcpInputProc - SSL rootCA=/opt/splunk/etc/certs/cacert.pem 09-21-2010 18:47:29.737 INFO TcpInputProc - SSL cipherSuite=ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM 09-21-2010 18:47:29.737 INFO TcpInputProc - supporting SSL v2/v3 09-21-2010 18:47:29.737 DEBUG TcpInputProc - SSL dhfile= 09-21-2010 18:47:29.737 DEBUG TcpInputProc - SSL requireClientCert=1 09-21-2010 18:47:29.740 INFO TcpInputProc - port 61025 is reserved for splunk 2 splunk (SSL) 09-21-2010 18:47:29.740 INFO TcpInputProc - Port 61025 is compressed 09-21-2010 18:47:29.740 DEBUG TcpInputProc - readConfig - creating acceptor for port 61025 09-21-2010 18:47:29.740 DEBUG TcpInputProc - Initing Acceptor with SSL 09-21-2010 18:47:29.740 INFO TcpInputProc - Registering metrics callback for: tcpin_connections (...then when the forwarder connects) 09-21-2010 18:51:27.812 INFO TcpInputProc - Connection in cooked mode from splunk-fwd-01.example.com 09-21-2010 18:51:27.816 INFO TcpInputProc - Valid signature found 09-21-2010 18:51:27.816 INFO TcpInputProc - Connection accepted from splunk-fwd-01.example.com on the forwarder (with category.TcpOutputProc=DEBUG in $SPLUNK_HOME/etc/log.cfg) : (During start up...) 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - groupName=[splunkssl] server=[192.168.1.100:9997] 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - creating group mapping for group splunkssl 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Validating URI - 192.168.1.100:9997 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Validation complete 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Setting : initialBackoff=2 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Setting : maxNumberOfRetriesAtHighestBackoff=-1 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Setting : maxBackoff=20 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Setting : backoffAtStartup=5 09-22-2010 16:22:12.017 INFO TcpOutputProc - Will retry at max backoff sleep forever 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - prop maxNumberOfRetriesAtHighestBackoff=-1 09-22-2010 16:22:12.017 INFO TcpOutputProc - Using SSL for server 10.1.12.1:61025, sslCertPath=/opt/splunk/etc/certs/forwarder.pem 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Key file password requires decrypting 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - prop sslRootCAPath=/opt/splunk/etc/certs/cacert.pem 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - prop sslVerifyServerCert=1 09-22-2010 16:22:12.017 INFO TcpOutputProc - Will verify Server's Certificate 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - prop sslCommonNameToCheck=splunk-idx-01.example.com 09-22-2010 16:22:12.017 INFO TcpOutputProc - Will check server's Common Name against splunk-idx-01.example.com 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - prop sslAltNameToCheck=splunk-idx-01 09-22-2010 16:22:12.017 INFO TcpOutputProc - ALL Connections will use SSL with sslCipher= 09-22-2010 16:22:12.018 INFO TcpOutputProc - initializing single connection with retry strategy for 192.168.1.100:9997 09-22-2010 16:22:12.018 DEBUG TcpOutputProc - creating sender thread for group ssltest 09-22-2010 16:22:12.018 DEBUG TcpOutputProc - registering timer callback 09-22-2010 16:22:12.019 DEBUG TcpOutputProc - IndexAndForward explicitly set to : false 09-22-2010 16:22:12.019 DEBUG TcpOutputProc - returning from updateConfig 09-22-2010 16:22:12.020 INFO TcpOutputProc - attempting to connect to 192.168.1.100:9997... 09-22-2010 16:22:12.020 DEBUG TcpOutputProc - Validating URI - 192.168.1.100:9997 09-22-2010 16:22:12.020 DEBUG TcpOutputProc - Validation complete 09-22-2010 16:22:12.029 INFO TcpOutputProc - Connected to 192.168.1.100:9997 09-22-2010 16:22:12.029 DEBUG TcpOutputProc - Sending signature 09-22-2010 16:22:12.029 DEBUG TcpOutputProc - Signature sent (...then, as the forwarder keeps the connection alive about every 30 seconds...) 09-22-2010 16:27:21.953 DEBUG TcpOutputProc - Inserting timer token into the queue. 09-22-2010 16:27:21.953 DEBUG TcpOutputProc - Processing timer event from the queue. 09-22-2010 16:27:21.953 DEBUG TcpOutputProc - Size of sockets is: 1 09-22-2010 16:27:52.958 DEBUG TcpOutputProc - Inserting timer token into the queue. 09-22-2010 16:27:52.958 DEBUG TcpOutputProc - Processing timer event from the queue. 09-22-2010 16:27:52.958 DEBUG TcpOutputProc - Size of sockets is: 1 7 - Troubleshooting : If things don't work out right! a) First, check in $SPLUNK_HOME/var/log/splunk/splunkd.log on both ends for errors. On the indexer, check for the messages from the TCP input processor "TcpInputProc", and on the forwarder, check the messages from the TCP output processor "TcpOutputProc". b) If necessary, increase the logging level of the appropriate processors on the indexer and the forwarder in $SPLUNK_HOME/etc/log.cfg. On the forwarder, set "category.TcpOutputProc=DEBUG", on the indexer set "category.TcpInputProc=DEBUG". Restart Splunk for these to take effect. c) Check the SSL configuration as it is seen by Splunk using btool. On the indexer : $SPLUNK_HOME/bin/splunk cmd btool inputs list --debug On the forwarder : $SPLUNK_HOME/bin/splunk cmd btool outputs list --debug d) Make sure that the certificates are readable by the user that Splunk runs as. Indexer-side, two common problems are : The path to the server certificate file set as the value of "serverCert" in inputs.conf is wrong, or the file cannot be read. This will generate the following error : 12-16-2010 16:07:30.965 ERROR SSLCommon - Can't read certificate file /opt/splunk/etc/auth/server.pem errno=33558530 error:02001002:system library:fopen:No such file or directory The password to the RSA private key contained in the server certificate file is wrong. This password is set as the value of "password" in inputs.conf. This will generate the following error : 12-07-2010 07:56:45.663 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem On *nix, you can manually test the password of the RSA key contained in the file by running the following openssl command : # openssl rsa -in /opt/splunk/etc/auth/server.pem -text The same can be done on Windows with the openssl binary that ships with Splunk : C:\Program Files\Splunk\bin>openssl.exe rsa -in "c:\Program Files\Splunk\etc\auth\server.pem" -text e) More information regarding the configuration of splunk2splunk SSL connections can be found here in the online documentation : http://www.splunk.com/base/Documentation/latest/Admin/EncryptandauthenticatedatawithSSL http://www.splunk.com/base/Documentation/latest/Admin/SecureaccesstoyourSplunkserverwithSSL The Splunk community wiki also has detailed tutorials on how to set up SSL for forwarding with 3 different scenarios : Configuring Splunk forwarding to use the default SSL server certificate Configuring Splunk forwarding to use SSL certificates self-signed by a newly generated root certificate Configuring Splunk forwarding to use SSL certificates signed by a third party Certificate Authority ... View more

During the first start up after the upgrade, Splunk tries to validate the existence of all declared indexes in the indexes.conf found in your configuration directories. Even if an index is disabled with "disabled = true" or by being present in the indexes.conf of a disabled app, Splunk will freak out if it cannot locate it's "homePath" (whether it's the default one in $SPLUNK_DB or a custom one specified in indexes.conf). The quickest way to troubleshoot this problem is to compare the output of two commands. The first one displays all existing index configuration stanzas across all indexes.conf : # find $SPLUNK_HOME/etc/ -name indexes.conf | xargs grep ^\\[ The second command will list the directories present in $SPLUNK_DB ($SPLUNK_HOME/var/lib/splunk/ by default), which is the default "homePath" for each index if no other value for that parameter is specified. # ls -l $SPLUNK_DB or # ls -l $SPLUNK_HOME/var/lib/splunk/ If the first command shows an index that is not present in $SPLUNK_DB and that index doesn't have a setting for "homePath" that points to an existing directory, you should consider disabling that index stanza in that configuration file by preceding it with a "#". Typical culprits are "splunklogger" (which has been deprecated and is often present but disabled) and "os" (which is often present in disabled apps like "unix" or "nixLF"). This is particularly effective on a forwarder where there shouldn't be any indexes configured other than what can be found in $SPLUNK_DB. Here's a typical scenario : 1) Which index directories exist? # ls -l $SPLUNK_DB total 52 drwx--x--x 5 root root 4096 Jun 25 16:29 audit drwx--x--x 2 root root 4096 Jun 25 10:14 authDb drwx--x--x 5 root root 4096 Jun 25 16:29 blockSignature drwx--x--x 5 root root 4096 Jun 25 16:29 defaultdb drwx--x--x 6 root root 4096 Jun 25 16:29 fishbucket drwx--x--x 2 root root 4096 Jun 25 16:29 hashDb drwx--x--x 5 root root 4096 Jun 25 16:29 historydb drwx--x--x 5 root root 4096 Jun 25 16:29 _internaldb drwx------ 2 root root 4096 Jun 25 16:29 persistentstorage drwx------ 2 root root 4096 Jun 25 16:27 queues drwx--x--x 5 root root 4096 Jun 25 16:29 sample drwx--x--x 5 root root 4096 Jun 25 16:29 splunkloggerdb drwx--x--x 5 root root 4096 Jun 25 16:29 summarydb 2) Which indexes are declared in the configuration files? # find $SPLUNK_HOME/etc/ -name indexes.conf | xargs grep ^\\[ etc/system/default/indexes.conf:[main] etc/system/default/indexes.conf:[history] etc/system/default/indexes.conf:[summary] etc/system/default/indexes.conf:[_internal] etc/system/default/indexes.conf:[_audit] etc/system/default/indexes.conf:[_thefishbucket] etc/system/default/indexes.conf:[_blocksignature] etc/system/default/indexes.conf:[splunklogger] etc/apps/sample_app/default/indexes.conf:[sample] etc/apps/nixLF/default/indexes.conf:[main] etc/apps/nixLF/default/indexes.conf:[history] etc/apps/nixLF/default/indexes.conf:[summary] etc/apps/nixLF/default/indexes.conf:[_internal] etc/apps/nixLF/default/indexes.conf:[_audit] etc/apps/nixLF/default/indexes.conf:[_blocksignature] etc/apps/nixLF/default/indexes.conf:[splunklogger] etc/apps/unix/default/indexes.conf:[os] etc/apps/SplunkLightForwarder/default/indexes.conf:[main] etc/apps/SplunkLightForwarder/default/indexes.conf:[history] etc/apps/SplunkLightForwarder/default/indexes.conf:[summary] etc/apps/SplunkLightForwarder/default/indexes.conf:[_internal] etc/apps/SplunkLightForwarder/default/indexes.conf:[_audit] etc/apps/SplunkLightForwarder/default/indexes.conf:[_blocksignature] etc/apps/SplunkLightForwarder/default/indexes.conf:[splunklogger] This revealed the following culprits : etc/apps/unix/default/indexes.conf:[os] etc/system/default/indexes.conf:[splunklogger] etc/apps/nixLF/default/indexes.conf:[splunklogger] etc/apps/SplunkLightForwarder/default/indexes.conf:[splunklogger] The "splunklogger" index was disabled everywhere it was declared with "disabled = true", and the "os" index was not in use as the unix app was disabled. Once these stanzas and all of their associated parameters were commented out, the upgrade worked! ... View more

If you intend to index the file instead of having Splunk monitoring/tailing it, then I think the CLI "oneshot" command is indeed the best solution. # splunk add oneshot <path_to_your_file> You could also create a new monitor input for that file from the CLI, this will index your file but it won't be at all the same than "uploading" it to Splunk as a one-time thing, as Splunk will be keeping track of it. # splunk add monitor <path_to_your_file> Another possibility is to toss the file in $SPLUNK_HOME/var/spool/splunk which is set up by default as a batch input (see $SPLUNK_HOME/etc/system/default/inputs.conf). Note that the file will be indexed destructively, so you may want to copy it there, not move it. ... View more

Provided that the data you are trying to consolidate in a single event comes from the same file input and is adjacent (i.e : lines following each other in the source file), what you want to do here is configure line-breaking to merge lines into a single event. The general instructions regarding line-breaking can be found here in our online documentation : http://www.splunk.com/base/Documentation/latest/Admin/Indexmulti-lineevents If you can include tags as delimiters for your events, this will make things easier and you can inform Splunk of this by populating LINE_BREAKER (in props.conf) with the adequate regex. From http://www.splunk.com/base/Documentation/latest/Admin/Propsconf : LINE_BREAKER = <regular expression> * Specifies a regex that determines how the raw text stream is broken into initial events, before line merging takes place. (See SHOULD_LINEMERGE) * Defaults to ([\r\n]+), meaning data is broken into an event for each line, delimited by \r or \n. * The regex must contain a matching group. * Wherever the regex matches, the start of the first matching group is considered the end of the previous event, and the end of the first matching group is considered the start of the next event. * The contents of the first matching group is ignored as event text. * NOTE: There is a significant speed boost by using the LINE_BREAKER to delimit multiline events, rather than using line merging to reassemble individual lines into events. There are other settings you may need to specify in your props.conf. Make sure SHOULD_LINEMERGE is set to true : SHOULD_LINEMERGE = true | false * When set to true, Splunk combines several lines of data into a single event, based on the following configuration attributes. * Defaults to true. If you are trying to include more than 256 lines in a single event, make sure that you tweak MAX_EVENTS and TRUNCATE accordingly : MAX_EVENTS = <integer> * Specifies the maximum number of input lines to add to any event. * Splunk breaks after the specified number of lines are read. * Defaults to 256. TRUNCATE = <non-negative integer> * Change the default maximum line length. * Set to 0 if you do not want truncation ever (very long lines are, however, often a sign of garbage data). * Defaults to 10000. ... View more

If your events always have those 20 fields separated by commas, you should use a delimiter-based field extraction, as described here in the Knowledge Manual : http://www.splunk.com/base/Documentation/latest/Knowledge/Createandmaintainsearch-timefieldextractionsthroughconfigurationfiles#Configuring_delimiter-based_field_extraction In your case, the following steps should work. 1 - Define the delimiter-based field extraction in $SPLUNK_HOME/etc/system/local/transforms.conf [delimiter_based_extraction] DELIMS = "," FIELDS = field1, field2, field3, field4, field5, field6, field7, field8, field9, field10, field11, field12, field13, field14, field15, field16, field17, field18, field19, field20 2 - Call that extraction for the desired spec (source, sourcetype or host) in $SPLUNK_HOME/etc/system/local/props.conf [my_sourcetype] REPORT-delims = delimiter_based_extraction The fields defined for extraction should now appear in the field picker. ... View more

The value of the "host" field for events of the "syslog" sourcetype is extracted using the "syslog-host" transform. The transform is called in $SPLUNK_HOME/etc/system/default/props.conf : [syslog] pulldown_type = true maxDist = 3 TIME_FORMAT = %b %d %H:%M:%S MAX_TIMESTAMP_LOOKAHEAD = 32 TRANSFORMS = syslog-host REPORT-syslog = syslog-extractions SHOULD_LINEMERGE = False The index-time filed extraction transform is defined $SPLUNK_HOME/etc/system/default/transforms.conf. The easiest way to prevent that extraction from happening is to suppress the call of the transform by populating the $SPLUNK_HOME/etc/system/local/props.conf file in the following way : [syslog] TRANSFORMS = This will result in the following effective configuration stanza for props.conf, which shows we have overwritten the TRANSFORMS used for the "syslog" sourcetype : # $SPLUNK_HOME/bin/splunk cmd btool props list syslog --debug system [syslog] system BREAK_ONLY_BEFORE = system BREAK_ONLY_BEFORE_DATE = True system CHARSET = UTF-8 system DATETIME_CONFIG = /etc/datetime.xml (...) system TIME_FORMAT = %b %d %H:%M:%S system TRANSFORMS = system TRUNCATE = 10000 system maxDist = 3 system pulldown_type = true After a restart of Splunk, the indexer will use the source ip/hostname of the forwarder that sent that data to populate the "host" field. ... View more

SplunkWeb's default time format display will be tied to the locale of the browser used to access the Splunk instance, as is described in the admin manual : http://www.splunk.com/base/Documentation/4.1.4/admin/Userlanguageandlocale#How_browser_locale_affects_timestamp_formatting This setting can be overridden by changing the URL you feed to your browser to access SplunkWeb, as described here : http://www.splunk.com/base/Documentation/4.1.4/admin/Userlanguageandlocale#Override_the_browser_locale For example, change the following URL : http://my-splunk-indexer:8000/en-US/app/search/dashboard ...to : http://my-splunk-indexer:8000/en-GB/app/search/dashboard ...in order to display the time stamp of your events in European format. Do note that at this time, Splunk only provides built-in support for a limited number of locales : en_GB en_US ja_JP zh_CN zh_TW ... View more

When using a whitelist or a blacklist in the definition of a [monitor] input, the total count of monitored files for that input in SplunkWeb doesn't subtract the files excluded by the white|blacklist. That count also includes directories traversed. This results in a very inflated number of monitored files displayed by SplunkWeb in Manager >> Data Inputs >> Files & Directories therefore doesn't reflect the actual number of monitored files. We currently (as of 4.1.4) have a bug opened against this behavior : SPL-32930 An accurate reading of the files Splunk is monitoring can be obtained by running : $SPLUNK_HOME/bin/splunk list monitor ... View more