topic Re: Splunk not reading the new file created after 2 months in Getting Data In

Splunk not reading the new file created after 2 months

ankithnageshshe — Thu, 28 Jun 2018 21:37:38 GMT

Hello Splunkers,

I have a situation where in a log file is created by the application after a long duration of 2 months.

I found no error in splunkd log for this specific file. Neither I found "WatchedFile" event for this file.
I'm sure that the issue is not due to initcrclen or crcSALT as the log file is new and splunkd log does not have any information on this.

After restarting the agent I finally get the following splunkd log info

06-28-2018 15:20:24.560 -0400 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='XXX.log'

However the old data is still not indexed and I do not have new data flowing in to the log file.

Can some one explain this situation.

Regards,
Ankith

Re: Splunk not reading the new file created after 2 months

adonio — Thu, 28 Jun 2018 23:49:15 GMT

can you share the inputs.conf?
do you have: ignoreOlderThan attribute there?

Re: Splunk not reading the new file created after 2 months

ankithnageshshe — Fri, 29 Jun 2018 13:35:57 GMT

Hello Adonio,

Thanks for your reply. Yes I have a parameter ignoreOlderThan= 14 days.
Is this the cause?

Re: Splunk not reading the new file created after 2 months

somesoni2 — Fri, 29 Jun 2018 15:01:07 GMT

Yes, with ignoreOlderThan, once a file is ignored from monitoring, it will stay ignored (won't be monitored) even if it gets some new data. When you restart Splunk, it re-evaluates the monitoring that needs to be done and will pick that file if it still newer than the ignoreOlderThan setting.

Re: Splunk not reading the new file created after 2 months

ankithnageshshe — Fri, 29 Jun 2018 16:02:14 GMT

Hi Somesoni2,

Thanks for the clarification. I'm trying to understand why splunk has not indexed the data even after the restart.
Situation: April 8th last log flow (after this file is rotated)
June 26th new log flow
Splunk doesnt perform watchedfile on this file.
After restart splunk performs watchedfile on this from the beginning of the file but the earlier data is not indexed.

Re: Splunk not reading the new file created after 2 months

somesoni2 — Fri, 29 Jun 2018 16:25:22 GMT

Are you monitoring the rolled log files? (check the [monitor:// in your inputs.conf). If you're not, those files will not be monitored/indexed.

Re: Splunk not reading the new file created after 2 months

woodcock — Sat, 30 Jun 2018 04:07:48 GMT

You need to adjust MAX_DAYS_AGO to cover your span, clear the fishbucket, and then restart the UF.

Re: Splunk not reading the new file created after 2 months

ankithnageshshe — Tue, 03 Jul 2018 13:44:32 GMT

Hello Woodcock,

Thanks for the reply.
I figured out that the issue is not due to the "ignoreolderthan" attribute as the issue appeared again.

It happened that splunk even skipped to watch one of the newly created file ( logs rotated every hour) without any error on the log file.

There is no configuration issue/ permission issue /port /network issue as other log file on the same path is read by the splunk. Also since rotation is 1 hour , ignoreolderthan attribute will not come in to picture.