topic Re: Splunk skips or delays indexing of the log file during the rotation occassionaly in Splunk Search

Splunk skips or delays indexing of the log file during the rotation occassionaly

ankithnageshshe — Tue, 14 Aug 2018 15:06:10 GMT

Hello Splunkers,

I have an issue where Splunk some times skips to index the log file during the rotation or delays the indexing during the log rotation.

This issue is only for specific file.So we can rule out the blocked queue, timezone, network throughput or slow performing indexer/forwarder.
Sar report showed good iostat cpu and mem stats on the forwarder.

I don't see initcrclength(crcSalT) or file_descriptor related issue in the splunk log.
In fact there are no error in the splunk log during this issue.

Any guidance is highly appreciated.

Best Regards,
Ankith

Re: Splunk skips or delays indexing of the log file during the rotation occassionaly

sudosplunk — Tue, 14 Aug 2018 17:33:14 GMT

Hi @ankithnageshshetty,

When you said log rotation, does the name of the file change? If so, then splunk doesn't monitor log rotations by default. However, you can adjust your inputs(.conf) to achieve this.

Re: Splunk skips or delays indexing of the log file during the rotation occassionaly

ankithnageshshe — Tue, 14 Aug 2018 17:57:20 GMT

Yes..the name of the file changes..
usually splunk continues to read the new file after the rotation. But some times splunk either skips the new file or delayes in indexing.

Re: Splunk skips or delays indexing of the log file during the rotation occassionaly

somesoni2 — Tue, 14 Aug 2018 19:35:03 GMT

How frequently the file roles? How much data does the file contains?

Re: Splunk skips or delays indexing of the log file during the rotation occassionaly

ankithnageshshe — Wed, 15 Aug 2018 14:58:38 GMT

Hello Somesoni,

File rotates every 30 minutes and size is 96MB.

Re: Splunk skips or delays indexing of the log file during the rotation occassionaly

somesoni2 — Wed, 15 Aug 2018 15:16:27 GMT

Depending upon the spike in data logging during those 30mins, there can be slowness in logs being ingested. Could you please provide monitoring stanza (inputs.conf) ? Also, how many rolled files do you keep and how are they renamed? (e.g. if myapp.log is name of monitored file, does it roll to myapp.log.1 first, then myapp.log.1 is renamed myapp.log.2 and next myapp.log is rolled as myapp.log.1 etc.)

Re: Splunk skips or delays indexing of the log file during the rotation occassionaly

ankithnageshshe — Wed, 15 Aug 2018 15:31:50 GMT

Hi somesoni,

Thanks for the prompt replies.

monitoring stanza:
[monitor:///app/logs/.../access]
sourcetype=ldap_access
index=XXX
ignoreOlderThan=14d

rotated file name: access.20180815152048Z . Basically it appends date and 6 digit number and "Z" to the file name.

1435 rotated files are present on the FS and retention is 56 days.

There is no information about file descriptor issue in the logs.

Re: Splunk skips or delays indexing of the log file during the rotation occassionaly

somesoni2 — Wed, 15 Aug 2018 15:40:49 GMT

So, in your monitoring stanza, the last access (.../access) is the file name Or directory? Ideally your monitoring stanza should be able to include rolled logs (you'd be monitoring both regularly written file and the rolled log file) as you're not using crcSalt=<SOURCE> (don't monitor rolled logs and use above crcSalt setting as it'll cause whole rolled log to be ingested again). With that, since Splunk is monitoring both regular log (will get each entry as they're new) and rolled logs (will not get everything else Splunk would recognize that it has already read those content, but will ingest anything that wasn't read).

Re: Splunk skips or delays indexing of the log file during the rotation occassionaly

ankithnageshshe — Wed, 15 Aug 2018 15:50:29 GMT

Hi somesoni,

access is file name and not the directory.

The actual file name is just "access" as mentioned in the monitoring stanza.

After the rotation new "access" file is created and old file is renamed to access.20180815152048Z

So I believe Splunk is only monitoring access here.

Re: Splunk skips or delays indexing of the log file during the rotation occassionaly

somesoni2 — Wed, 15 Aug 2018 16:43:58 GMT

Ok.. So try this for your monitoring stanza. You should see improvement. (restart after making change if making change directly)

[monitor:///app/logs/.../access*]
sourcetype=ldap_access
index=XXX
ignoreOlderThan=14d

Re: Splunk skips or delays indexing of the log file during the rotation occassionaly

ankithnageshshe — Wed, 15 Aug 2018 18:58:58 GMT

Will this not read the rotated file? creating duplicate indexing?

Re: Splunk skips or delays indexing of the log file during the rotation occassionaly

jcrabb_splunk — Tue, 29 Sep 2020 20:55:41 GMT

There is a bug for this in various versions. When the file rotates, Splunk stops reading the file until the file rotates again. At that point it ingests both files (catches up). The work around is to configure time_before_close = 1 under the relevant input.

[monitor://<path>]
time_before_close = 1

If you are on 6.6.x, this is fixed in 6.6.4:

http://docs.splunk.com/Documentation/Splunk/6.6.4/ReleaseNotes/6.6.4#Data_input_issues

SPL-142334, SPL-143553, SPL-145370, SPL-145978 logs are delayed in reading after rotation

This particular version of the bug is also fixed in:

7.1.0 (SPL-143553)
7.0.1 (SPL-145978)

I have seen this in 6.4.x as well and the provided work around (listed above) resolved the issue.

Re: Splunk skips or delays indexing of the log file during the rotation occassionaly

ankithnageshshe — Thu, 16 Aug 2018 13:45:17 GMT

Thanks jcrabb for the update.

Re: Splunk skips or delays indexing of the log file during the rotation occassionaly

vinaykata — Tue, 29 Sep 2020 22:16:00 GMT

Hi, I am having the same issue with IIS logs and right now we are in 6.6.3. We are not able to upgrade our versions sooner because of some other issues we are having. So my question is, is this issue resolved for 6.6.3 or is it just above 6.6.4. we have tried adding "time_before_close = 10" in our inputs.conf, but we did not see any improvements or changes. Thanks everyone!!!

Re: Splunk skips or delays indexing of the log file during the rotation occassionaly

gjanders — Wed, 05 Dec 2018 21:51:11 GMT

Fixed in 6.6.4 and the other versions mentioned by jcrabb, you will need to upgrade to a newer version to avoid the issue

Re: Splunk skips or delays indexing of the log file during the rotation occassionaly

knielsen — Wed, 30 Sep 2020 03:06:40 GMT

Is it possible that bugfix never made it to the 7.2 tree?

We experience this issue on forwarder 7.2.6, and setting time_before_close to 1 seems to help so far.