It really depends on your platform, Unix has several utilities that give you information on this - iostat, dtrace, vmstat etc - this page contains useful information on all of these. The Unix app for Splunk contains 'iostat' and 'vmstat' scripted inputs, and you can configure/create any other ones you would like to visualize For Windows, you will likely need to install some custom tool as there's not really an equivalent to the Unix commands above, but the Splunk for Windows app will give you other useful information. ... View more

'Cannot create' suggests a permissions issue, but not having much expertise with pkgask, that's just a guess. A lot of Customers have had success with Puppet to roll Splunk out to a high number of servers, and our in-house PS guys can probably offer advice on any other tools they have used. This is really where your local System Administrator should be brought into the conversation, because if you don't have a process or tool already in place to do something like this, then it's the best time to choose/create one. This won't be just a one-time activity, as updates and new releases will be coming out continuously and when you're dealing with hundreds of servers, you need to track and maintain the details of what bits are being installed on each server. ... View more

So 4.1 has introduced the new tailing processor, which is now capable of handling many more files than the old version - wahoo! The above info message just means that Splunk has more files to read but it has already reached the limit of open FD's. 3.x versions of Splunk used the max_fd setting in limits.conf so you could tune Splunk to increase this overall number, but that has not yet been added to the 4.1.x code-branch. It's coming though, and will be in the official 4.1.3 release, which is next on the list. For now, if you're monitoring directories that contain 800 files, it's not going to be much of an issue and its just a case of Splunk cleaning up open FD's so it can move on to the next file. It will always come back and read your files again, so it shouldn't be missing out on any data. ... View more

You can delete the existing fishbucket contents with the command detailed here - ./splunk clean eventdata _thefishbucket Splunk now trackes the files it has seen and indexed/forwarded in the splunk_private_db directory in $SPLUNK_DB/fishbucket so as long as the 'many gigs' of data isn't inside this directory, we can safely remove it. ... View more

The answer is actually yes, you should be able to use white & blacklist settings for sinkhole directories (batch inputs). The underlying logic is the same for both monitor and batch inputs, the only difference being that batch is destructive and will delete your data. I'll get the docs updated to reflect this. ... View more

The problem here is that by default, Splunk limits the amount of disk-space a user can use while searching. This is to ensure that all resources are not consumed by one user running lots of inefficient searches, and it is documented here. This page will also tell you how to increase the quota in authorize.conf for a specific role. Searches run in a seperate process to splunkd, so just because splunkd is killed/restarted doesn't necessarily mean that the searches themselves will also disappear. If you want to get a view into what searches are running, you can click on the Jobs link in the top right corner and see all the current and recent searches by a user. Any jobs that persist on disk will take up space, and I believe the default lifetime of a search object is 15 minutes. If you clear some space by deleting the jobs, the messages will cease. The message being displayed is produced when a User tries kick off a new search. Splunk will first check the disk quota in use from previous searches to ensure that it has enough space to store results in $SPLUNK_HOME/var/run/splunk. If you run 'du -sh *' on that directory, you'll be able to identify the searches that are using so much space and manually remove them. It's likely that the directories no longer exist at this point unless you have set the retention time pretty high. In the current version of 4.1, there is no 'self-monitoring' capability for searches, so they will continue to run and eat up disk space until finished. The disk-quota check is run just once at the beginning of the search to ensure the space exists and the search can be run. If the check finds that the quota has been reached, the new search will NOT run but previous large searches will not be finalized automatically. Also, real-time searches have specific windows and buffers to ensure that they don't take up excessive disk-space so we don't expect them to be an issue in this case, just the historic searches. In the upcoming version, 4.2, searches will be able to monitor themselves and auto-finalize when the quota is reached and notifications will be pushed to Users via the UI and logs. ... View more

Both packages should install on any of the OS versions listed. However, I do recommend that you rethink your platform choice for your Splunk server. The SPARC architecture is well known to be a performance inhibitor for Splunk, so if you want your instance to be able to search and index data at a high rate, you should deploy on a 64-bit Linux or Windows platform if available. ... View more

You can disable the login page banner content by changing the updateCheckerBaseURL setting in $SPLUNK_HOME/etc/system/local/web.conf updateCheckerBaseURL = http://quickdraw.Splunk.com/js/ * These are various Splunk.com urls that are configurable. * Setting updateCheckerBaseURL to 0 will stop the SplunkWeb from pinging Splunk.com for new versions of itself. The only way to remove the version information at the bottom of the page would be to edit the template files in $SPLUNK_HOME/share/splunk/search_mrsparkle/templates/account/login.html and .../logout.html - however, these updates will not persist through an upgrade so you will have to re-apply them as part of your upgrade process ... View more

We went about trying to recreate this behaviour, and decided that the only way those errors could occur is if the permissions on the 'Manager' pages themselves aren't present. Those pages actually live in the 'search' app and the permissions are controlled by files in $SPLUNK_HOME/etc/apps/search/metadata . If you looked at your diag, you may see that you are either missing the default.meta file and only have a local.meta file in there or, the default.meta file is present but the content has been altered somehow. default.meta is the file that exports the page permissions to the other apps. If you changed the 'search' app via the Deployment server, tha might be an explanation as to why it disappeared, as it should be present when Splunk is first installed. To restore it, just extract the file from the package tarball, copy it to the correct location and restart. ... View more