Security

Splunk skips to watch newly created file occasionally

ankithnageshshe
Path Finder

Hello All,

I have forwarders where in some of the log files are rotated on a hourly basis and some of them are created less often (once in 2 weeks may be) based on the log flow .

I observe that splunk skips to watch the newly created log files occasionally and does not index the log.

I can confirm that there is no configuration issue/ permission issue/network issue as this happens occasionally . Most of the log files are read by the splunk as soon as the file is created after log rotation.

I only have ignoreolderthan set to 14 days in inputs.conf and I can confirm that this could not be the issue as hourly rotated log files are also not read by splunk at time.

There is no error /info relating to watching the newly created file in the splunkd.log

Restarting the splunk forwarder will make the splunk to watch these skipped files although old data is somehow not indexed.

I would wish to know the reason behind this splunk behavior.

Thanks in advance.

Regards,
Ankith

Tags (1)

skoelpin
SplunkTrust
SplunkTrust

How similar is your data? The fishbucket works by looking at the first and last part of your log file to identify if its already indexed the data to prevent duplicates. If your data is very similar then it could be tricking your fishbucket into thinking the file has already been indexed. Adding crcSalt to your inputs.conf would help fix this issue

crcSalt = <string>
* Use this setting to force the input to consume files that have matching CRCs
  (cyclic redundancy checks).
    * (The input only performs CRC checks against, by default, the first 256
      bytes of a file. This behavior prevents the input from indexing the same
      file twice, even though you may have renamed it -- as, for example, with
      rolling log files. However, because the CRC is based on only the first
      few lines of the file, it is possible for legitimately different files
      to have matching CRCs, particularly if they have identical headers.)
* If set, <string> is added to the CRC.
* If set to the literal string <SOURCE> (including the angle brackets), the
  full directory path to the source file is added to the CRC. This ensures
  that each file being monitored has a unique CRC.   When crcSalt is invoked,
  it is usually set to <SOURCE>.
* Be cautious about using this setting with rolling log files; it could lead
  to the log file being re-indexed after it has rolled.
* In many situations, initCrcLength can be used to achieve the same goals.
* Defaults to empty.

http://docs.splunk.com/Documentation/Splunk/latest/admin/Inputsconf

0 Karma

ankithnageshshe
Path Finder

Hi Skoelpin,

Thanks for your reply. Data is not similar in any of the file ( connection id is unique for each log generated).
Also I don't see any log information for crcSALT or initcrclenght.
It seems that splunk was not able to read the new file created and it was pointing to the old rotated file.

When I restart the forwarder it says "seekptr did not match..will continue to read the entire file from the begining."
Also I dont see "begin reading from offset=0" log for the affected files in splunkd.log

0 Karma

skoelpin
SplunkTrust
SplunkTrust

What version of the UF are you running? You should open a support case

0 Karma

ankithnageshshe
Path Finder

I'm running Splunk UF 6.4...I dont see any information on splunkd for not reading the log files.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...