Why have monitored inputs stopped working?

[djreschke]

Has anyone run into an issue where a Splunk HF, is not monioring files being written to it. This HF is also a syslog server so files are being written to it and the monirotied inputs are on the server. The file ingestion happens after a restart. Any pointers?

[jamie00171]

Hi @djreschke 

Do the first 256 bytes of the file happen to be the same on each file in the same input? 

If so, take a look at the following from (https://docs.splunk.com/Documentation/Splunk/9.0.0/Admin/inputsconf😞

crcSalt = <string>
* Use this setting to force the input to consume files that have matching CRCs,
  or cyclic redundancy checks.
    * By default, the input only performs CRC checks against the first 256
      bytes of a file. This behavior prevents the input from indexing the same
      file twice, even though you might have renamed it, as with rolling log
      files, for example. Because the CRC is based on only the first
      few lines of the file, it is possible for legitimately different files
      to have matching CRCs, particularly if they have identical headers.
* If set, <string> is added to the CRC.
* If set to the literal string "<SOURCE>" (including the angle brackets), the
  full directory path to the source file is added to the CRC. This ensures
  that each file being monitored has a unique CRC. When 'crcSalt' is invoked,
  it is usually set to <SOURCE>.
* Be cautious about using this setting with rolling log files; it could lead
  to the log file being re-indexed after it has rolled.
* In many situations, 'initCrcLength' can be used to achieve the same goals.
* Default: empty string

Thanks, 

Jamie

[sirpatrick]

Perhaps related but I have Universal Forwarders that are monitoring a small number of files being written by a dedicated syslog server, but today I noticed some monitored are being processed but others are not. No errors that I have found in the logs, Splunkd.log shows it was processing fine, then didn't show any tail reader entries for the missing files. This is on a Windows server which was patched this weekend and the monitored files are being written.

[richgalloway]

If file ingestion happens after a restart then when does it stop?  How many files are being monitored?  Are ulimits set correctly?

Is the HF acting as the syslog server or is it just monitoring files written by a dedicated syslog server (like syslog-ng)?  If the former, why?  That's bad practice.  If the latter, consider using a universal forwarder unless transforms are needed.

[djreschke]

@richgalloway The time it stops is unknown at this time, I get alerted when logs stop being collected on a daily basis. Its a syslog server so a lot of files, 20 - 40 files depending on activity. ulimit is set at 0. 

[richgalloway]

20-40 files is not a lot for Splunk to monitor.

I'm concerned about ulimit settings of zero.  Splunk recommends higher values than that.  See https://docs.splunk.com/Documentation/Splunk/latest/Installation/Systemrequirements#Considerations_r...

A lower ulimit can have unknown effects on Splunk, possibly including not monitoring files.

[djreschke]

@richgalloway I agree, it is not a lot of files. Current ulimit settings

# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 514862
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 102400
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 514862
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

[djreschke]

@richgalloway limits.conf in the local folder 

limits.conf
[thruput]
maxKBps = 00