This is the fallout from CHECK_FOR_HEADER. There's a bug where CHECK_FOR_HEADER is applied based on filename pattern, despite not being in your sourcetype. You might be able to work around by declaring CHECK_FOR_HEADER = false explicitly in your sourcetype, or by declaring [source::.../Failed Attempts.*.csv] sourcetype=acs in props.conf, especially if you set a priority value of eg 100. The bug is due for fixing in 4.1.4, which will be a few weeks -- we're verifying bugs as fast as we can. ... View more

Is this "How can i get splunk to identify the time of an event by a timestamp in seconds from epoch?" If that's the question, the answer is TIME_FORMAT=%s TIME_PREFIX=Start_Time_ms= Err update: what I gave will pull seconds from epoch, not milliseconds. I don't know if our extractor will give the right answer for milliseconds. It might just pull the first n digits, but could be off for very old timestamps. I haven't tested. ... View more

This is SPL-31174, there is a small flaw in the migration from 4.0 to 4.1. For 4.1, the 'field extractions' manager UI page was modified into a sub-page of the 'fields' manager page. However, the old page is accidentally left in place. To succesfully create and manage fields, you can simply click on the Fields link instead of Field Extractions. To remove the bogus UI element, you can remove or rename the page element splunk/etc/apps/search/default/data/ui/manager/data_extractions.xml Note, this problem should not occur in Splunk 4.1.5 and later. ... View more

The 4.1.x implementation of file input leans heavily on stat() to get information about which files should be opened and read. For network inputs, it's quite possible to simply have total latency for a very large number of stat()s become unreasonably large. I understand you're seeing the nfs system is not overloaded, but what sort of latency are we seeing? I'm not familiar with troubleshooting with nfsstat. I would think to look at the i/o picture with iostat and/or using strace -p to look at the system call time of the stat()s, and then extrapolate. What is your total number of files in the monitored hierarchies? Essentially, a number something like this: find /the /directories /you /monitor |wc -l The specifics of the monitor lines might be useful, as well as any information about subdirectories of these you might not actually be interested in. ... View more

Firstly, for many cases, applying sourcetypes by file pattern can be preferable to content-based recognition. The file pattern rules are easier to audit. This can be done via overlapping input stanzas (in 4.1+), or by source:: regex-like patterns in props.conf. If you do have the need to apply sourcetypes by content, then the train command is busted (SPL-31078), but it's just a candy wrapper over the following, which you can use directly for full effect: $SPLUNK_HOME/bin/splunk cmd classify path/to/myfile mysourcetype If you're trying to use train to recognize timestamps, I generally recommend using TIME_FORMAT instead. ... View more