Getting Data In

Latest log not showing in Splunk

justindett
Path Finder

Hi,

I have a weird issue where when a log rolls and a new log gets created, it takes about a day or so to actually show the new log in Splunk. Looking on the server, the new log exists. But Splunk is only showing the last log before the new one was created.

Any idea why this would happen?

Thanks

0 Karma
1 Solution

justindett
Path Finder

Enabling the crcSalt seemed to have solved the issue. Logs seem to be up to date for the last couple days now.

Thanks for all the suggestions

View solution in original post

0 Karma

justindett
Path Finder

Enabling the crcSalt seemed to have solved the issue. Logs seem to be up to date for the last couple days now.

Thanks for all the suggestions

0 Karma

woodcock
Esteemed Legend

Probably when the log rolls, the new log is created with the wrong ownership or permissions so that user splunk cannot read it but then there is a housekeeping ( probably cron-based) job that comes around once a day and deleted old files and fixes ownership and permissions. This should be easy to check, just keep doing this until you see it rotate and look:

ls -altr /Your/Path/To/Files/Here
0 Karma

woodcock
Esteemed Legend

You probably have too many co-resident files. At hundreds of files (whether or not Splunk is supposed to forward them or not, or whether it already has or not), things slow down (like you are seeing). At thousands of files, things pretty much completely stop. A good test is that if you get a significant surge just after restarting the forwarder and then it goes back to really, really slow, then this is your problem. Do proper OS-level housekeeping to move/archive/delete older files and things will go back to snappy again.

0 Karma

justindett
Path Finder

@woodcock There are only 30 logs in this directory. I have enabled the crcSalt now as well. Lets see if that makes a difference.

Sahr_Lebbie
Path Finder

That was going to be my suggestion(crcSalt). How did it work out for you?

When you say renamed, were there new log file names being created or were files moving to a new directory and the same log file being appended to but just new logs?

0 Karma

justindett
Path Finder

I found a similar issue here : https://answers.splunk.com/answers/680732/splunk-skips-or-delays-indexing-of-the-log-file-du.html

Made the change as specified : time_before_close = 1

But doesn't look like it helped. Forwarder version is 7.0.3

Unless I need to wait until the log rolls again at midnight tonight?

0 Karma

justindett
Path Finder

Below is the content of the inputs.conf The whole log directory is specified, but its always just picked up the original .log file which is fine.

[monitor:///WebSphere8/applications/dev/psiberworks/logs]
disabled = false
whitelist = .log$

crcSalt = SOURCE

index = ibm_was_app_psi-was8-dev-01
time_before_close = 1

0 Karma

DavidHourani
Super Champion

What happens if your crcSalt is enabled ? do you still have the issue ?

0 Karma

DavidHourani
Super Champion

Hi @justindett,

Which files are you using for your input ? The original one or the rolled one ?

0 Karma
Get Updates on the Splunk Community!

Thanks for the Memories! Splunk University, .conf24, and Community Connections

Thank you to everyone in the Splunk Community who joined us for .conf24 – starting with Splunk University and ...

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...