Our central syslog server forwards syslog data to my Splunk server, using TCP (secure syslog). In the Splunk web GUI, each line is prepended with a number contained in angle brackets (<>), like this: <166>Feb 28 17:52:31 192.168.22.192 httpd[27331]: www.example.org 192.168.33.18 - - [28/Feb/2012:17:52:30 -0800] "GET /somepage HTTP/1.0" 200 6838 "-" "gsa-crawler" <132>Feb 28 17:52:31 192.168.33.182 slapd[12312] LDAP message <38>Feb 28 17:52:31 192.168.72.67 DROP packet on firewall X What do the numbers in angle brackets mean? Are they related to syslog facilities and priorities? This syslog data is also forwarded to another syslog server which does not run Splunk. ... View more

I am using Splunk with SSO (Shibboleth) for authentication. Unfortunately, I still need to create a Splunk user for every user coming in through SSO. How can I bulk import users into Splunk, preferably from the command line? There is a file named $SPLUNK_HOME/etc/passwd . I have added users to this file, but they don't appear in the web interface under http://splunk.example.org/en-US/manager/launcher/authentication/users . In addition, Splunk removes users from this file periodically. The command $SPLUNK_HOME/bin/splunk import userdata -dir /tmp/export.dat , but there is almost no documentation about this feature, and I cannot find anything which describes the format of export.dat . It looks as if this feature is really intended to export userdata from Splunk and import it to another Splunk instance, which is not what I am trying to do. ... View more

This is a Scientific Linux 6.1 system (Equivalent to RHEL 6.1). I installed Splunk as the root user, and it starts and runs fine as root. Now I want to run Splunk as the user 'splunk' instead of as the user 'root'. So I follow the instructions at RunSplunkasadifferentornon-rootuser. That page says (in bold) Then, before you start Splunk for the first time, change the ownership of the splunk directory to the desired user. But in this case, I have already installed Splunk as root and have already started Splunk as root. # useradd splunk # groupadd splunk # chown -R splunk:splunk $SPLUNK_HOME # ls -ld $SPLUNK_HOME drwxr-xr-x 9 splunk splunk 4096 Dec 8 13:10 /data/splunk And then I try to start splunk, as the user splunk . The following command is from /etc/init.d/splunk . Splunkd can start, but splunkweb fails to start. # /bin/su splunk -c "/data/splunk/bin/splunk start" Splunk> The IT Search Engine. Checking prerequisites... Checking http port [8000]: open Checking mgmt port [8089]: open Checking configuration... Done. Checking index directory... Validated databases: _audit _blocksignature _internal _perf_report _perf_test _thefishbucket history main splunkit_idxtest summary Done Success Checking conf files for typos... All preliminary checks passed. Starting splunk server daemon (splunkd)... [ OK ] Error starting splunkweb. [FAILED] Done.Starting splunkweb... I looked in $SPLUNK_HOME/var/log/splunk/ (web_service.log splunkd.log) and I see absolutely no indication of failure. The splunkweb service failed without writing any information to those logs. I ran an strace, but can't figure out what is failing: # su - splunk -c "/usr/bin/strace /data/splunk/bin/splunk start splunkweb" ... ... stat("/data/splunk/etc/auth/splunkweb", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0 stat("/data/splunk/etc/auth/splunkweb/cert.pem", {st_mode=S_IFREG|0600, st_size=802, ...}) = 0 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7ffa303b39d0) = 5391 wait4(5391, [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0, NULL) = 5391 --- SIGCHLD (Child exited) @ 0 (0) --- stat("/etc/rc.d/init.d/functions", {st_mode=S_IFREG|0644, st_size=17921, ...}) = 0 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7ffa303b39d0) = 5393 wait4(5393, Error starting splunkweb. [FAILED] [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0, NULL) = 5393 --- SIGCHLD (Child exited) @ 0 (0) --- write(1, "Starting splunkweb... ", 22Starting splunkweb... ) = 22 exit_group(1) Why does splunkweb fail to start? ... View more

I have a Scientific Linux 6.1 server. I want Splunk to start upon boot, and I want to start/stop/restart Splunk using standard utilities like /sbin/service . What is the proper way to use chkconfig with the Splunk binaries? Does Splunk provide a standardized startup script that I can place in /etc/init.d/ ? I cannot find an answer to this under http://docs.splunk.com/Documentation/Splunk It appears that Splunk supports chkconfig. If I do strings splunk I see that the binary does contain the string chkconfig . ... View more

I'm evaluating Splunk 4.1 (build 77833) on my Mac laptop. Splunk has been turned off since April. When I started Splunk, the application said something like "You should upgrade... security vulnerabilities, something something". I have since lost that notification box, and it looks like it had some useful links on it. Is it possible to get that back? I see that a version 4.1.3 is available from the download page. I see many warnings about using care when upgrading. However, I don't see any instructions on how to upgrade from my version to 4.1.3, here or at READTHISFIRST Is there anything special that I need to do before upgrading from Splunk 4.1 (build 77833) to ... View more