Why does my Splunk universal forwarder monitor stop processing files the next day after they roll over?

[a212830]

Hi,

I have a Splunk Universal Forwarder running on Windows 2012, monitoring a bunch of files in different folders. The files are monitored fine, until the next day, when they stop. No idea why. The files are in the format someUniqueIdentifier_hostname_MM-DD-YYYY_HHMMSS.txt.

Here are some examples:

VMHostConfig_ABCC002VWIN.FMR.COM_01-01-2016_012804.txt
VMHostConfig_ABCC002VWIN.FMR.COM_01-02-2016_012801.txt

My inputs looks like this:

[monitor://D:\productionscriptdata\vmhost_config\VMHostConfig_*.txt]
index = distributed
sourcetype = vmhostconfig
followTail = 0
crcSalt = <SOURCE>
ignoreOlderThan = 7d

Any suggestions?

[prakash007]

Check your splunkd.logs and metrics.log if you can find any errors.

[sloshburch]

Are the files older than the 7d limit? I know that's too easy 😞
Was there anything in the splunkd.log corresponding to that monitor input? You can turn on debug mode for that component (http://docs.splunk.com/Documentation/Splunk/6.3.2/Troubleshooting/Enabledebuglogging ), or restart splunk on that forwarder with debug mode by running ./splunk restart --debug (although I think it no longer exists for windows forwarders - i forget)

What technology generates the new files? Might be worth seeing if other folks have had issues with that and splunk?

Some may suggest the alwaysOpenFile setting but I say wait until support has you implement that because of the performance issues.

That said, if you can't find evidence of this, nor any blocking within metrics.log, you should consider a support ticket as it sounds like the feature is not working as documented/expected.

[esix_splunk]

Do these files all happen to have the same headers? Sometimes if the header length is over the default of 256bytes, Splunk won't recognize a rolled file. Check the inputs.conf spec for :

initCrcLength = <integer>

• This setting adjusts how much of a file Splunk reads before trying to identify whether it is a file that has already been seen. You may want to adjust this if you have many files with common headers (comment headers, long CSV headers, etc) and recurring filenames.

Change that to a larger value..

[a212830]

No, no headers. It's all unique, with timestamps.