There is a very severe issue with forwarding to a 3rd party syslog from any full Splunk instance  (SH, HF, Indexer, etc.). Let' say you want to send data to a 3rd party syslog via TCP and it happens that syslog is unreachable (not receiving data for whatever reason (service down, network down, etc.). After some time all queues start to fill up from output (e.g. output queue for syslog target group is just 97B and you cannot change it) over parsing (parsing, merging, typing, ...) to input queues. Once input queue is full Splunk instance will stop receving data - on indexer you effectively STOP indexing. This is by design. There are some solutions for this - none is great.  1. Send to a 3rd party syslog via UDP - not reliable and for many customer unacceptable. 2. Send requrested content via some other "tool". If you want to send out locally monitored files you can use SyslogNG or Cribl if the target destination is syslog-capable consumer.  BTW Cribl is the only non-Splunk solution out on the market that supports native input/output for Splunk proprietary S2S protocol. So you can collect data by UF and send to Cribl for processing and forward further to Splunk, Syslog  and many other targets. You cannot solve this puzzle with Splunk only ideally. ... View more

That was it, thanks for the help adamw! ... View more

Hi, We're currently using Splunk on Solaris with ZFS as the underlying filesystem. As you may know, ZFS offers filesystem base compression and we can also choose between different compression algorythms.. Since it usually is a bad idea to try to compress data twice, I was looking for a way to disable Splunk's index compression so I can validate if there are gains to be made by using ZFS compression. This being said, I also think it could be useful for people using NAS solutions that are based on ZFS (Sun's ZFS appliance, Nexenta's solutions, SGI's nas, etc). This would let the storage device deals with compression and leave as much cpu as possible for the indexer. ... View more

I love it thanks! ... View more

We don't use SMTP auth in our environment, so I can't speak to the specifics regarding that, but I can say that we use an FQDN (splunk@mydomain.com) here, and it works without issue. It does look like some have had success configuring the email server stuff in the actual search query itself, but I'm not sure if that helps... http://splunk-base.splunk.com/answers/38624/how-to-configure-email-alert-using-gmail-smtp ... View more

Hi, Do we need to restart the splunk instances for the passwords to work? ... View more

I had the same issue for a csv log file with a header. My fix was to add the following line to my monitor rule in inputs.conf on the Universal Forwarder. crcSalt= <SOURCE> http://docs.splunk.com/Documentation/Splunk/latest/admin/inputsconf crcSalt = <string> * Use this setting to force Splunk to consume files that have matching CRCs (cyclic redundancy checks). (Splunk only performs CRC checks against the first few lines of a file. This behavior prevents Splunk from indexing the same file twice, even though you may have renamed it -- as, for example, with rolling log files. However, because the CRC is based on only the first few lines of the file, it is possible for legitimately different files to have matching CRCs, particularly if they have identical headers.) * If set, <string> is added to the CRC. * If set to the literal string <SOURCE> (including the angle brackets), the full directory path to the source file is added to the CRC. This ensures that each file being monitored has a unique CRC. When crcSalt is invoked, it is usually set to <SOURCE>. * Be cautious about using this attribute with rolling log files; it could lead to the log file being re-indexed after it has rolled. * Defaults to empty. ... View more

Thank you! Problem was in props.conf in how I was referencing the transform. All is working now! ... View more

Splunk takes a little while to spin up after a restart, how long are you waiting for these logs to show up after a restart? Also depending on how you have the input set up, you could potentially speed up the process. If you have [monitor:/var/log/*] Splunk will (on restart) iterate through the entire /var/log/ directory, listing all files in the directory and all subdirectories, then it will begin traversing down a directory level and on and on until it has finished. Then it will start indexing the files. ... View more

When configuring the splunk deployment client, you can set clientName = in deploymentclient.conf as the hostname of the particular client to be sent to the deployment server. This would allow you to create stanzas in serverclass.conf to web-1.domain.com, even though the actual hostname on this particular server might be web.domain.com Thanks, --adam ... View more

might be Splunk will auto-tune itself to go with the highest priority fields ?! not sure... But, i hope that Splunk wont delete the data , it makes the index size to grow till max size maxTotalDataSizeMB frozenTimePeriodInSecs ... View more

$SPLUNK_DB/persistentstorage (also for fschange). ... View more

I agree with piebob, we've found in our deployment that adding indexers and doing load balancing and distributed search significantly improves index and search performance. Basically search heads are CPU intensive, while indexers are memory intensive. So plan accordingly, but scaling indexers will give you your most bang for your buck, imho. ... View more

Hi all, any updates on this? Still the same with Splunk v6.4.4 Regards, Andreas ... View more

Are you noticing that only processes run by SYSTEM show up? We are noticing this in our environment. ... View more

hostnames were not properly configured on alerts_actions.conf file that failed to display results for email link results, This was corrected by adding stanza in /opt/splunk/etc/system/local/alert_action.conf: [email] hostname = ... View more

I agree, a rogue DATETIME_CONFIG = CURRENT entry does seem to be the most logical explication for what's going on here. ... View more

You should not have had to do this. I have overridden views by copying and modifying them in local. The default folder will be restored after applying a service release. ... View more

I'm sensing an insurmountable amount of pain from using summary indexing to accomplish this, plus I was wondering if it was something simple and I just hadn't realized it yet. Thanks! ... View more

What do you mean by you want fschange to "diff" the files? Do you mean you want it recorded whenever the contents change, or do you mean you want the contents of every file version recorded? (Basically, do you want fullEvent=false, or fullEvent=true on fschange?) If it's the latter, I don't think you need to do anything. When you want to diff the file content, you can simply use wildcards in the source name when you query for it, rather than a specific file name. If you want, you could create a TRANSFORM to modify the source name into a "base" source that doesn't contain a timestamp, so you'd search on source=config rather than source=config.* . If it's the latter and you don't want unchanged versions recorded, I think you could use monitor instead of fschange, and in props.conf set CHECK_METHOD = entire_md5 for the file. ... View more

I too have similar problem. where i have deployment server, which is a unix system and deployment client as windows. File permission on unix system for 'rename_app' is drwxrwxrwx . When i deploy the 'rename_app' from unix system, i am getting 'access issue' for 'rename_app' folder. inputs.conf : [script://$SPLUNK_HOME\etc\apps\rename_app\bin\rnminput.bat] interval = 30 disabled = false If the issue is file permission in window, how to set the permission of window folder/file from unix server. Please advise. Thanks! ... View more

Have you tried removing: [syslog] defaultGroup=logserver from your outputs.conf? As I read it and from my own experience it might change the behaviour from what you intend. Otherwise it looks good. If all else fails you can use REGEX = to negate certain log entries. ... View more