Our Splunk setup holds its hot and warm buckets on local disks, cold buckets are on FC storage. What would happen if the FC storage becomes unavailable, e.g. crashed, unmounted, unresponsive? Would Splunk try to roll the buckets to cold and become unresponsive itself? Would Splunk hold off the rolling to cold and fill local warm buckets until FC storage becomes available? Regards, Mitja ... View more

There are about 1600 files that are actively monitored by the forwarder (64bit, v5.0.2) on our central syslog server. The forwarder process constantly hogs one whole core. Where to start looking for possible problems and solutions? Regards, Mitja ... View more

I am running Splunk version 5.0.2. I want to add write permissions for the navigation menu to a user role. I go to "Manager » User interface » Navigation menus" as the admin user and click the Permissions link for the entry "default". I check the appropriate user role in the Write column and click Save. The file /opt/splunk/etc/apps/search/metadata/local.meta show this for the navigation menu file: [nav/default] access = read : [ * ], write : [ admin, naviwrite ] owner = nobody version = xxxx modtime = xxxx Users in naviwrite can't edit/save the content of the navigation menu entry "default". When I click on the Permissions link again, the only role ticked in the Write column is admin. The content of local.meta still shows access = read : [ * ], write : [ admin, naviwrite ] The test setup does not have these problems. Looks like local.meta is totally ignored. How can I get this to work? ... View more

Log messages received by our central loghost take up to 2 hours before being visible on the indexer. Our network hardware and servers send their messages via syslog to a central loghost running syslog-ng which filters the messages into their respective files. That totals to about 1600 log files with current data. The log files rotate daily at midnight (their name is in the form of service-20130410.log). Then there is a universal forwarder, that monitors the folders/file for changes and forwards that data to a Splunk indexer. Altogether we index about 12GB of data per day. A typical monitor stanza looks like this: [monitor:///logs/splunk/servicetype] host_segment=4 index = main ignoreOlderThan = 3d sourcetype = servicetype The indexer has a certain search that runs every hour and requires certain data from the past hour. The results of the search at 1AM and 2AM come up empty every day. Ther were also instances of the 3AM search comming up empty. After that point all subsequent searches return data as they should. There is no more delay. I have checked the indexer and it should not be the bottleneck. Disk I/O, CPU (24 core), and RAM (32GB) should not be a problem on the loghost server. Although the UF is constantly hogging 1 core to the maximum. There is a delay between the time files are created and when the universal forwarder notices and forwards them. How can I tune this to speed this up? Is the 1600 monitored files considered a high or a low number for the universal forwarder? Kind regards, Mitja ... View more

We have some old indexed events that have their host field value set to the name of the forwarder on the central syslog host. It appears, that hostname extraction was not configured correctly at that time. I do not wish to delete and re-index these events since the only thing "wrong" is the hostname. Here is a sample message: Nov 12 00:00:00 sampleHostname processname[1642]: 6 [yaddayaddaID] The message that follows I would like to override the hostname field value at search time so instead of forwarder.example.com it will show sampleHostname. If this is at all possible. ... View more