topic Re: Latest log not showing in Splunk in Getting Data In

Latest log not showing in Splunk

justindett — Thu, 21 Nov 2019 06:45:49 GMT

Hi,

I have a weird issue where when a log rolls and a new log gets created, it takes about a day or so to actually show the new log in Splunk. Looking on the server, the new log exists. But Splunk is only showing the last log before the new one was created.

Any idea why this would happen?

Thanks

Re: Latest log not showing in Splunk

justindett — Wed, 30 Sep 2020 03:06:45 GMT

Made the change as specified : time_before_close = 1

But doesn't look like it helped. Forwarder version is 7.0.3

Unless I need to wait until the log rolls again at midnight tonight?

Re: Latest log not showing in Splunk

DavidHourani — Thu, 21 Nov 2019 13:24:32 GMT

Hi @justindett,

Which files are you using for your input ? The original one or the rolled one ?

Re: Latest log not showing in Splunk

justindett — Wed, 30 Sep 2020 03:06:59 GMT

Below is the content of the inputs.conf The whole log directory is specified, but its always just picked up the original .log file which is fine.

[monitor:///WebSphere8/applications/dev/psiberworks/logs]
disabled = false
whitelist = .log$

crcSalt = SOURCE

index = ibm_was_app_psi-was8-dev-01
time_before_close = 1

Re: Latest log not showing in Splunk

DavidHourani — Thu, 21 Nov 2019 14:07:06 GMT

What happens if your crcSalt is enabled ? do you still have the issue ?

Re: Latest log not showing in Splunk

woodcock — Thu, 21 Nov 2019 15:09:56 GMT

You probably have too many co-resident files. At hundreds of files (whether or not Splunk is supposed to forward them or not, or whether it already has or not), things slow down (like you are seeing). At thousands of files, things pretty much completely stop. A good test is that if you get a significant surge just after restarting the forwarder and then it goes back to really, really slow, then this is your problem. Do proper OS-level housekeeping to move/archive/delete older files and things will go back to snappy again.

Re: Latest log not showing in Splunk

justindett — Thu, 21 Nov 2019 16:46:18 GMT

@woodcock There are only 30 logs in this directory. I have enabled the crcSalt now as well. Lets see if that makes a difference.

Re: Latest log not showing in Splunk

Sahr_Lebbie — Sat, 23 Nov 2019 01:52:34 GMT

That was going to be my suggestion(crcSalt). How did it work out for you?

When you say renamed, were there new log file names being created or were files moving to a new directory and the same log file being appended to but just new logs?

Re: Latest log not showing in Splunk

woodcock — Sat, 23 Nov 2019 16:43:34 GMT

Probably when the log rolls, the new log is created with the wrong ownership or permissions so that user splunk cannot read it but then there is a housekeeping ( probably cron-based) job that comes around once a day and deleted old files and fixes ownership and permissions. This should be easy to check, just keep doing this until you see it rotate and look:

ls -altr /Your/Path/To/Files/Here

Re: Latest log not showing in Splunk

justindett — Mon, 25 Nov 2019 06:16:40 GMT

Enabling the crcSalt seemed to have solved the issue. Logs seem to be up to date for the last couple days now.

Thanks for all the suggestions