topic Re: Is alwaysOpenFile still available in version 4.1? in Getting Data In

Is alwaysOpenFile still available in version 4.1?

Lowell — Tue, 06 Apr 2010 20:34:42 GMT

Does anyone know if alwaysOpenFile still works in inputs.conf as of Splunk 4.1. It still shows up in the 4.1 docs, but there is a note saying that it will be removed in 4.1, which is confusing.

Quoted from http://www.splunk.com/base/Documentation/latest/Admin/Inputsconf:

alwaysOpenFile = 0 | 1
* NOTE: This setting does not exist in 4.1.  Look for it in an upcoming version.
* Opens a file to check if it has already been indexed.
* Only useful for files that don't update modtime.
* Should only be used for monitoring files on Windows, and mostly for IIS logs.
* NOTE: This flag should only be used as a last resort, as it increases load and slows down indexing.

We use this flags for some logs generated on a Windows system that don't get indexed in version 3.4.x without this flag. We are looking to soon upgrade to 4.0 or 4.1. If splunk has been improved to automatically detect and handle this better, that's great, it just seem like the docs aren't clear on this specific point.

Re: Is alwaysOpenFile still available in version 4.1?

jrodman — Thu, 08 Apr 2010 23:19:54 GMT

We expect that alwaysOpenFile will not be needed.

The usual usecase was to work around situations where the file modification time was not an accurate indicator of file content changes. In 4.1 our file update notification mechanism (FUN), is interested in both file modtime changes, as well as file size changes. It doesn't really even care if the file time is after the last check, just that it changed since the last check.

On Windows, the common problem was an ancient bug that Microsoft considers a feature, where some methods of writing to files do not update the mod time until file close, defeating the purpose of modification time utterly. Since file size is now used as one of the indicators, this common case will not require any special flags.

As a result, we suspect that there is no particular need for this setting in 4.1, but a new situation may arise. Note that this would have to be a case where the file size does not change, nor does the file modification time, but the contents do.

Re: Is alwaysOpenFile still available in version 4.1?

jrodman — Fri, 09 Apr 2010 04:57:30 GMT

Amrit points out there's a case where this isn't complete. Since IIS and friends increase filesize 64k at a time, for relatively quiescent servers this flag might still be useful to get the data more in realtime.

Re: Is alwaysOpenFile still available in version 4.1?

ftk — Tue, 13 Apr 2010 00:54:03 GMT

Wanted to write a comment to jrodman's answer, but no link to do it.

After updating to 4.1, all files that I had to use alwaysOpentFile=1 on do no longer get indexed with latest updates. Mod time doesn't change, but the file size definitely does -- it still doesn't get indexed though.

I am specifically talking about logs from RRAS (Routing and Remote Access Service) as well as the Windows port of snmptrapd's log files. YMMV, but I recommend to set up an eval install first and see if your files get indexed before upgrading.

[EDIT] After a splunk restart (physical machine due to updates) and switching RRAS log rotation from monthly to daily, Splunk seems to index the files fine now. I am keeping an eye on the inputstatus and will notify appropriate parties in case I notice the problem again. For now I am chalking it up to a one time fluke.

Re: Is alwaysOpenFile still available in version 4.1?

Lowell — Tue, 13 Apr 2010 01:07:44 GMT

Are you suggesting that you are unable to index these files with or without the alwaysOpenFile setting in 4.1? (Yeah, I'm doing an eval now. I don't think I'd ever upgrade to a x.y.0 release without some testing time first.). BTW, you have alwaysOpentFile NOT alwaysOpenFile (perhaps just a typo, but just in case you copied it from a config file, thought you should know about it.)

Re: Is alwaysOpenFile still available in version 4.1?

ftk — Tue, 13 Apr 2010 01:24:37 GMT

@Lowell, just a typo -- I mean alwaysOpenFile. Sorry about that. And yeah, at this point I am unable to index updates to log files from RRAS and Windows snmptrapd since upgrading to 4.1.

Re: Is alwaysOpenFile still available in version 4.1?

ftk — Tue, 13 Apr 2010 01:47:12 GMT

@Lowell after a splunk restart and switching the RRAS log rotation from monthly to daily it appears to index fine now
-- may have just been a one time glitch. I am keeping an eye on it.