windows file locking?

[bfaber]

can Splunk read an windows share when files are still be written to it?

[Michael_Wilde · Accepted solution]

Sure, if the file isn't locked by an application. Windows has a much different way of locking files that UNIX (which has the concept of "inodes"). It can lock them up so tight that you can't even open them read-only. For databases, this is a good idea. For logfiles, this is inconvenient. A good way to test whether splunk will be able to monitor/index a file while its being written is to open it with Notepad. If Notepad can't open it, Splunk probably won't be able to. But there are some exceptions. Microsoft Exchange's Message Tracking Log. We've seen Splunk have no problem indexing it, but a user couldn't open it in notepad... why?

When a windows app creates an object for accessing/writing a file with the CreateFile() class, there are some options that it can open/create that file with. Those options are around sharing access to the file.

http://msdn.microsoft.com/en-us/library/aa363874.aspx

I'm just going to guess... that Exchange uses CreateFile() and has sharing set to "FILE_SHARE_READ". I'm also going to guess when notepad uses CreateFile() --which is also used for opening a file, it tries to get GENERIC_WRITE privs on a file...

As engineering has verified for me, when Splunk uses CreateFile() to access a file, it first tells Windows it would like "GENERIC_READ" on the file and options that basically allow reading, writing, or deleting by other apps. Splunk is going to do its best to open the file, not lock it in anyway, but if other apps need to open that file as well, no problem.

If i am right about how Notepad opens a file with GENERIC_WRITE, then thats why is why Splunk can read it.. when Notepad cant.

More about file access: http://msdn.microsoft.com/en-us/library/aa364399(VS.85).aspx

In most cases where Splunk cannot read a logfile it is sadly a failure of the app developer who has chosen to not share read access to the log file while its being written.

[jrodman]

The short version is the logging application has to open the file with FILE_SHARE_READ, and if it doesn't, there's nothing anyone else can do to read the file, including Splunk.