There is a log file I want want monitor with splunk universal forwarder (on windows) which receives line writes only during a phase of windows startup before the splunk universal forwarder is running..
I am able to successfully search line changes when subscribing to file-based inputs using the monitor input type. My only concern with this approach is that it appears that Splunk polls the file system for changes to file every second, which seems like wasted disk i/o.
I tried the MonitorNoHandle input type but that does not seem to work, I assume because the forwarder is not running at point in time when line changes occur.
Is there a way to control the interval of polling for file changes with the monitor input type? Is there an alternate method (short of script-based input) you would recommend? I can also just "get over" the wasted i/o line of thought. 🙂
I was digging through documentation and saw this
interval attribute that "If you set the interval to a negative number, Splunk Enterprise runs the input one time."
This is under the
WinHostMon stanza, so I'm not sure if it'll apply to the
monitor stanza input type, but it might be a setting worth testing out? When I checked http://docs.splunk.com/Documentation/Splunk/6.2.1/admin/inputsconf for the
interval setting under
WinHostMon, it wasn't as detailed as the table included in the first docs link, so maybe there's hope with testing that setting. When it says "runs the input one time", I'm also not sure if that means one time for all-time, or each time Splunk Enterprise (UF) and/or Windows machine is restarted. Otherwise, seems like the interval can just be set to another number so it doesn't poll every second.
Thanks for digging and clever idea! I just gave that a shot but no dice!
Despite the change, sysinternals procmon still exposes persistent polling of the file by splunkd every second.
How long did you watch it with procmon? The forwarder is supposed to back off and poll files less frequently when the files themselves are not being frequently updated. If you find this is not true, it may be worth a support case to find out why.
But, one thing to think about is that the OS will do a good job of caching these files. A once a second poll may seem like a lot, but it really doesn't add an appreciable amount of I/O to your system unless we are talking about millions of files. The kernel will keep this file in cache, and you won't do much actual I/O against it.
I didn’t realize it would back-off over time. That would be a good option. So far I am 30+ minutes in from last file write but splunkd is still polling on 1s intervals. I will let this run for 24h and report back if the polling rate changes.
This type of file I/O is not a big deal to me for operating system instances having exclusive access to disk. My reservations come with use among virtualized systems sharing disk.
I'll take a lower level look at this to see what the implications to disk are... cause..learning and stuff.