Hello, The Splunk App for Unix and Linux has a limit of 2000 hosts. ... View more

Double check your Account and/or Input names do not have any spaces in them. If they do, delete and recreate without spaces and you should be able to get this certificate to generate successfully. ... View more

Take a look at the python_modular_input.log file. This will give you clues as to what is happening. In this case, a field called _time was created and that is not allowed. I saw this error message: ERROR pid=7888 tid=identity file=lookup_modinput.py:streaming_merge_task:283 | status="Exception when reading input files" exc=Field names cannot start with an underscore: '_time' Once the field was corrected to not start with an underscore, things worked fine. ... View more

The problem here is because the multikv.conf file that is located in the Splunk_TA_aws does not exist on the indexer. In this case, it was only installed on the search head. To remove the error, either install the Splunk_TA_aws on the indexer or copy the multikv.conf file located in etc/apps/Splunk_TA_aws/default from the search head to the indexer and then restart the splunk service. ... View more

The problem here is that splunk was configured with boot start enabled. On AIX, this will put a line in the /etc/inittab file which will then be run by startsrc. This seems fine except startsrc is run by root and if your Splunk install is configured to run as a different user, you can get a "permission denied" error. To fix this, remove the line in /etc/inittab that references Splunk and restart. If you do want Splunk to start at boot time, you could configure it with the "enable boot start" but just make sure to set the user Splunk will run as correctly. This doc page is a good reference. http://docs.splunk.com/Documentation/Splunk/6.1.3/admin/ConfigureSplunktostartatboottime ... View more

Run this command from your splunk /bin directory: ./splunk cmd python -V You'll see output similar to this: Python 2.7.5 ... View more

These files are used to make a backup copy of the fishbucket. The error is indicating that the user Splunk is running as, doesn't have permission to the folder path listed. To correct it, you'll need to correct the permissions listed for your user running Splunk. This should stop this error message from happening. ... View more

You observation is correct. This behavior is identified as Bug in Splunk Version 6. Bug # SPL-74975.Currently it is expected to be fixed in release 6.0.2 ... View more

One thing to make sure of (especially if upgrading from a much earlier version of Splunk) is that you are not set to search in Fast mode - http://docs.splunk.com/Documentation/Splunk/6.0/Search/Changethesearchmode Note this point especially - Fast Mode Disables field discovery. Field discovery is the process Splunk uses to extract fields aside from default fields such as host, source, and sourcetype. This means that Splunk only returns information on default fields and fields that are required to fulfill your search (if you are searching on certain fields, it will extract those fields). Try searching in verbose mode and see if your extractions now appear. ... View more

In this situation, go and check the $SPLUNK_HOME/etc/instance.cfg. What happened in this case was one server was cloned and the second box had the same GUID value in the instance.cfg. You can delete the GUID value in the instance.cfg, save, restart Splunk, and you'll be good to go. ... View more

There was a rogue nic card that came online at some point after the upgrade a couple of weeks ago and picked up a secondary ip address and became the preferred path. However this rogue address was not routable across the network and hence the intermittent connections to the indexers. We could not detect this because the main interface was still working fine when connecting to the server remotely. Thanks for mentioning physical location, that led me to logon to the server again and check the network interfaces and route table and found the problem. ... View more

The answer is yes, but of course this depends on your requirements. The simiplist answer is to use oneshot using the CLI. You can issue one shot remotly on any splunk instance so long as you have enable remote access or have change the default password for the admin account. I have on occasion scripted a remote oneshot to index files for one off cases. If your looking to reset your fishbucket btprob is the real answer. To reset one file in your fishbucket from your UF: ./btprobe -d SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db --file /var/log/access.log --reset Note: Your splunk daemon or service must be stopped and if you try to use the base directory you will recieve "could not open '/var/log/' (Access is denied.)". If you doing this on a UF the btprobe will need to be copied to your UF. You should recieve the following message: key=0x7b100f52c71e67f1 scrc=0x6e16f6c553b73581 sptr=8070178 fcrc=0xcdcd2890474a56fb flen=0 mdtm=1347001048 wrtm=1352329761**95Record (key 0x7b100f52c71e67f1) reset. If in my case I have hundreds of test file sitting on a UF in mulitple directories, so I wrote a perl script to feed each file in each directory to the btprobe utility. Hope this helps you or gets you started. Dont forget to accept or thumbs up answers. Additional Reading: MonitorfilesanddirectoriesusingtheCLI CommandlinetoolsforusewithSupport Hope this help or gets you started. ... View more

In all likelihood, this is happening because the log writer is buffering content before appending it to the file you are monitoring and occasionally pausing mid-event for longer than the default 'time_before_close' duration (3s), as defined in inputs.conf.spec: time_before_close = <integer> * Modtime delta required before Splunk can close a file on EOF. * Tells the system not to close files that have been updated in past <integer> seconds. * Defaults to 3. The solution here is to adjust 'time_before_close' (on a per-stanza basis in inputs.conf) so that Splunk's tailing processor will only close a given file after the longest pause that the log writer can take. For example, if you know that the log writer can pause for up to 5 seconds mid-event, adjust 'time_before_close' to a value that can comfortably accomodate that, say 10. ... View more