Getting Data In

Why does my Splunk universal forwarder monitor stop processing files the next day after they roll over?

a212830
Champion

Hi,

I have a Splunk Universal Forwarder running on Windows 2012, monitoring a bunch of files in different folders. The files are monitored fine, until the next day, when they stop. No idea why. The files are in the format someUniqueIdentifier_hostname_MM-DD-YYYY_HHMMSS.txt.

Here are some examples:

VMHostConfig_ABCC002VWIN.FMR.COM_01-01-2016_012804.txt
VMHostConfig_ABCC002VWIN.FMR.COM_01-02-2016_012801.txt

My inputs looks like this:

[monitor://D:\productionscriptdata\vmhost_config\VMHostConfig_*.txt]
index = distributed
sourcetype = vmhostconfig
followTail = 0
crcSalt = <SOURCE>
ignoreOlderThan = 7d

Any suggestions?

0 Karma

prakash007
Builder

Check your splunkd.logs and metrics.log if you can find any errors.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Are the files older than the 7d limit? I know that's too easy 😞
Was there anything in the splunkd.log corresponding to that monitor input? You can turn on debug mode for that component (http://docs.splunk.com/Documentation/Splunk/6.3.2/Troubleshooting/Enabledebuglogging ), or restart splunk on that forwarder with debug mode by running ./splunk restart --debug (although I think it no longer exists for windows forwarders - i forget)

What technology generates the new files? Might be worth seeing if other folks have had issues with that and splunk?

Some may suggest the alwaysOpenFile setting but I say wait until support has you implement that because of the performance issues.

That said, if you can't find evidence of this, nor any blocking within metrics.log, you should consider a support ticket as it sounds like the feature is not working as documented/expected.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Do these files all happen to have the same headers? Sometimes if the header length is over the default of 256bytes, Splunk won't recognize a rolled file. Check the inputs.conf spec for :

initCrcLength = <integer>
  • This setting adjusts how much of a file Splunk reads before trying to identify whether it is a file that has already been seen. You may want to adjust this if you have many files with common headers (comment headers, long CSV headers, etc) and recurring filenames.

Change that to a larger value..

0 Karma

a212830
Champion

No, no headers. It's all unique, with timestamps.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...